Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

How CLIP Turns Text Prompts into a Zero-Shot Image Classifier

AI summary

A Juejin AI tutorial explains that CLIP performs zero-shot image classification by encoding candidate classes as text prompts and selecting the prompt whose embedding is most similar to the image embedding. It walks through image and text encoding, L2 normalization, cosine similarity, temperature scaling and Softmax, then provides OpenCLIP code for basic and ensemble-based classification. The article also warns that outputs depend on the candidate set, class names and prompt templates, and that closed-set selection can force an unsuitable match.

In-depth insight

The guide offers a practical implementation of CLIP’s zero-shot workflow while making clear that prompts and candidate labels form an integral part of the classifier rather than cosmetic inputs.

一句话理解:CLIP 并没有学习一个固定的“猫狗分类器”,而是把类别写成文本,再判断当前图像与哪段文本最匹配。

上一篇《CLIP 的训练机制:从对比学习到图文对齐》介绍了 CLIP 的训练过程:图像经过 Image Encoder,文本经过 Text Encoder,模型通过对比学习把配对的图文拉近、把不匹配的图文推远。

但训练完成之后,这个模型究竟怎么用于分类?

传统分类模型通常需要预先确定类别,并为这些类别训练一个固定的分类头。CLIP 却可以只看类别名称,直接识别训练阶段没有专门标注过的新任务。这种能力被称为 Zero-shot Classification(零样本分类)。

这一篇,我们就从推理过程出发,看看一组普通的文本描述,是如何临时变成图像分类器的。

一、什么是零样本分类?

假设现在有一张图片,我们希望模型判断它属于下面哪个类别:

#CLIP#OpenCLIP#OpenAI

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.