CLIP

We support OpenAI's CLIP model for generating image and text embeddings, and for zero-shot similarity comparison between them, via our Serverless Cloud API. We expose three endpoints:

• /clip/embed_image, returns an embedding vector for an image
• /clip/embed_text, returns an embedding vector for a string or list of strings
• /clip/compare, returns similarity scores between a subject and a list of prompts

Embeddings can be cached and reused for tasks like classification, retrieval, clustering, and semantic search. For broader usage details, see the Inference documentation.

CLIP API

Below is a code sample that compares an image against a list of text labels. Call the HTTP endpoint directly with curl, or use the inference-sdk wrapper.

export ROBOFLOW_API_KEY="your-key-here"

curl --location 'https://serverless.roboflow.com/clip/compare' \
  --header 'Content-Type: application/json' \
  --data '{
    "api_key": "'"$ROBOFLOW_API_KEY"'",
    "subject": {"type": "url", "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"},
    "subject_type": "image",
    "prompt": ["a photo of a dog", "a photo of a cat", "a photo of a car"],
    "prompt_type": "text"
  }'

export ROBOFLOW_API_KEY="your-key-here"

pip install -U inference-sdk supervision

import os
import supervision as sv
from inference_sdk import InferenceHTTPClient

image = sv.load_image_from_url("https://media.roboflow.com/notebooks/examples/dog.jpeg")

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["ROBOFLOW_API_KEY"],
)

result = client.clip_compare(
    subject=image,
    prompt=[
        "a photo of a dog",
        "a photo of a cat",
        "a photo of a car",
    ],
    subject_type="image",
    prompt_type="text",
)

# similarity is a list of cosine similarity scores, one per prompt
print(result["similarity"])

[0.2726989686489105, 0.19865083694458008, 0.20997387170791626]

CLIP inference speed

Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.

Measured with embed_image on the ViT-B-16 checkpoint (image embedding only).

Run CLIP with self-hosted Inference

CLIP also runs entirely on your own hardware with the inference Python package. Loading the weights in-process avoids a network round trip per call, which matters when you are embedding large image sets for search, clustering, or dataset cleaning.

pip install "inference[clip]"

from inference.models import Clip
from inference.core.utils.postprocess import cosine_similarity

clip = Clip(model_id="clip/ViT-B-16")

image_embedding = clip.embed_image("https://media.roboflow.com/inference/people-walking.jpg")
text_embedding = clip.embed_text("a crowd of people walking")

print(cosine_similarity(image_embedding[0], text_embedding[0]))

Available checkpoints

model_id selects the CLIP backbone:

clip/RN50, clip/RN101, clip/RN50x4, clip/RN50x16, clip/RN50x64, clip/ViT-B-32, clip/ViT-B-16, clip/ViT-L-14, clip/ViT-L-14-336px.

The SDK methods clip_compare, get_clip_image_embeddings, and get_clip_text_embeddings accept a clip_version argument to select the same checkpoints when calling a server.

Further reading

• What is CLIP?
• Build an image search engine with CLIP and Faiss
• Analyze and classify video with CLIP