SAM3

We support Meta's Segment Anything Model 3 inferencing via our Serverless Cloud API. We offer two different SAM3 endpoints:

• Promptable concept segmentation (PCS), which segments every instance of a concept in the image. Concepts are described by text prompts, exemplar boxes, or both.
• Promptable visual segmentation (PVS), which interactively segments one object per request from points or a box, in the style of SAM2.

Use this table to pick an endpoint:

Pass your API key as the api_key query parameter on every request.

SAM3 Concept Segmentation (PCS)

POST https://serverless.roboflow.com/sam3/concept_segment

Each entry in prompts describes one concept. The response contains one prompt_results entry per prompt, each holding every instance found. Requests accept at most 16 prompts.

Text prompts

import os
import requests

payload = {
    "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"},
    "prompts": [
        {"type": "text", "text": "person"},
        {"type": "text", "text": "backpack"},
    ],
    "output_prob_thresh": 0.5,
    "format": "polygon",  # or "rle"
}

response = requests.post(
    "https://serverless.roboflow.com/sam3/concept_segment",
    params={"api_key": os.environ["ROBOFLOW_API_KEY"]},
    json=payload,
)
for prompt_result in response.json()["prompt_results"]:
    print(prompt_result["echo"], len(prompt_result["predictions"]), "instances")

Images can also be sent inline as {"type": "base64", "value": "<BASE64_IMAGE>"}.

Exemplar box prompts

Instead of text, you can prompt with an exemplar: a box around one example object. The model finds every instance that matches the example, not just the boxed object.

payload = {
    "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"},
    "prompts": [
        {
            "type": "visual",
            "boxes": [{"x": 1409, "y": 705, "width": 112, "height": 183}],
            "box_labels": [1],
        }
    ],
    "output_prob_thresh": 0.5,
    "format": "polygon",
}

Boxes use absolute pixel coordinates. Two formats are accepted:

• {"x": ..., "y": ..., "width": ..., "height": ...} where x, y is the top-left corner
• {"x0": ..., "y0": ..., "x1": ..., "y1": ...} for explicit corners

box_labels is required when boxes is set and must have one entry per box: 1 marks a positive exemplar (find objects like this), 0 marks a negative exemplar (exclude objects like this).

Combined text and exemplar prompts

A single prompt can carry both text and exemplar boxes. This is useful for narrowing a text concept with visual examples, or excluding lookalikes with negative exemplars:

payload = {
    "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"},
    "prompts": [
        {
            "type": "visual",
            "text": "person",
            "boxes": [
                {"x": 1409, "y": 705, "width": 112, "height": 183},
                {"x": 1216, "y": 496, "width": 124, "height": 184},
            ],
            "box_labels": [1, 0],
        }
    ],
    "output_prob_thresh": 0.5,
    "format": "polygon",
}

Here the model segments people matching the first (positive) exemplar while suppressing instances similar to the second (negative) exemplar.

SAM3 Visual Segmentation (PVS)

POST https://serverless.roboflow.com/sam3/visual_segment

PVS segments one specific object indicated by clicks or a box. Use it for interactive, human-in-the-loop mask refinement; use PCS when you want every instance of a concept.

import os
import requests

payload = {
    "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"},
    "prompts": {
        "prompts": [
            {
                "points": [{"x": 1465, "y": 796, "positive": True}],
                "box": {"x": 1465, "y": 796, "width": 112, "height": 183},
            }
        ]
    },
    "multimask_output": False,
    "format": "json",
}

response = requests.post(
    "https://serverless.roboflow.com/sam3/visual_segment",
    params={"api_key": os.environ["ROBOFLOW_API_KEY"]},
    json=payload,
)
prediction = response.json()["predictions"][0]
print(prediction["confidence"], len(prediction["masks"]), "polygons")

A prompt can contain points, a box, or both:

• points are absolute pixel coordinates. "positive": true includes the clicked region, false excludes it. Add more points to refine the mask.
• box uses center-anchored coordinates: x, y is the box center, unlike PCS boxes which are top-left anchored.

The response contains the single highest-confidence mask for the prompt. multimask_output controls how many internal mask proposals the model generates (three when true), but the best proposal is always selected for the response.

For an interactive demo using OpenCV, see this GitHub Gist, which was used in this video:

SAM3 inference speed

Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.

Measured with concept segmentation from a single text prompt.

SAM3 API endpoints

POST /sam3/concept_segment?api_key=text HTTP/1.1
Host: serverless.roboflow.com
Content-Type: application/json
Accept: application/json

{
  "image": {
    "type": "url",
    "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"
  },
  "prompts": [
    {
      "type": "text",
      "text": "person"
    },
    {
      "type": "text",
      "text": "car"
    }
  ],
  "output_prob_thresh": 0.5,
  "format": "polygon"
}

curl -L \
  --request POST \
  --url 'https://serverless.roboflow.com/sam3/concept_segment?api_key=text' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
  "image": {
    "type": "url",
    "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"
  },
  "prompts": [
    {
      "type": "text",
      "text": "person"
    },
    {
      "type": "text",
      "text": "car"
    }
  ],
  "output_prob_thresh": 0.5,
  "format": "polygon"
}'

const response = await fetch("https://serverless.roboflow.com/sam3/concept_segment?api_key=text", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Accept": "application/json"
  },
  body: JSON.stringify({
    "image": {
      "type": "url",
      "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"
    },
    "prompts": [
      {
        "type": "text",
        "text": "person"
      },
      {
        "type": "text",
        "text": "car"
      }
    ],
    "output_prob_thresh": 0.5,
    "format": "polygon"
  })
});

const data = await response.json();
console.log(data);

import requests

url = "https://serverless.roboflow.com/sam3/concept_segment?api_key=text"
headers = {
    "Content-Type": "application/json",
    "Accept": "application/json"
}
payload = {
  "image": {
    "type": "url",
    "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"
  },
  "prompts": [
    {
      "type": "text",
      "text": "person"
    },
    {
      "type": "text",
      "text": "car"
    }
  ],
  "output_prob_thresh": 0.5,
  "format": "polygon"
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

{
  "prompt_results": [
    {
      "prompt_index": 0,
      "echo": {
        "prompt_index": 0,
        "type": "text",
        "text": "dog",
        "num_boxes": 0
      },
      "predictions": [
        {
          "masks": [
            [
              [
                345,
                251
              ],
              [
                344,
                252
              ],
              [
                343,
                253
              ]
            ]
          ],
          "confidence": 0.89453125,
          "format": "polygon"
        }
      ]
    }
  ],
  "time": 0.221
}

{
  "detail": [
    {
      "loc": [
        "anything"
      ],
      "msg": "text",
      "type": "text"
    }
  ]
}

POST /sam3/visual_segment?api_key=text HTTP/1.1
Host: serverless.roboflow.com
Content-Type: application/json
Accept: application/json

{
  "image": {
    "type": "url",
    "value": "http://www.example-image-url.com"
  },
  "image_id": "image_id",
  "format": "json",
  "sam2_version_id": "hiera_large",
  "multimask_output": true,
  "save_logits_to_cache": false,
  "load_logits_from_cache": false
}

curl -L \
  --request POST \
  --url 'https://serverless.roboflow.com/sam3/visual_segment?api_key=text' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
  "image": {
    "type": "url",
    "value": "http://www.example-image-url.com"
  },
  "image_id": "image_id",
  "format": "json",
  "sam2_version_id": "hiera_large",
  "multimask_output": true,
  "save_logits_to_cache": false,
  "load_logits_from_cache": false
}'

const response = await fetch("https://serverless.roboflow.com/sam3/visual_segment?api_key=text", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Accept": "application/json"
  },
  body: JSON.stringify({
    "image": {
      "type": "url",
      "value": "http://www.example-image-url.com"
    },
    "image_id": "image_id",
    "format": "json",
    "sam2_version_id": "hiera_large",
    "multimask_output": true,
    "save_logits_to_cache": false,
    "load_logits_from_cache": false
  })
});

const data = await response.json();
console.log(data);

import requests

url = "https://serverless.roboflow.com/sam3/visual_segment?api_key=text"
headers = {
    "Content-Type": "application/json",
    "Accept": "application/json"
}
payload = {
  "image": {
    "type": "url",
    "value": "http://www.example-image-url.com"
  },
  "image_id": "image_id",
  "format": "json",
  "sam2_version_id": "hiera_large",
  "multimask_output": True,
  "save_logits_to_cache": False,
  "load_logits_from_cache": False
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

{
  "prompt_results": [
    {
      "prompt_index": 1,
      "predictions": "anything"
    }
  ],
  "time": 1
}

{
  "detail": [
    {
      "loc": [
        "anything"
      ],
      "msg": "text",
      "type": "text"
    }
  ]
}

Run SAM3 with self-hosted Inference

SAM3 can also run on your own hardware, either loaded in-process with the inference package or served from a GPU container.

Run in Docker

docker run -it --rm -p 9001:9001 --gpus=all roboflow/inference-server:latest

The server exposes the same /sam3/concept_segment and /sam3/visual_segment endpoints documented above at http://localhost:9001.

Load the model in Python

pip install "inference-gpu[sam3]"

import os

os.environ["API_KEY"] = "YOUR_API_KEY"

from inference.core.entities.requests.sam3 import Sam3Prompt
from inference.models.sam3 import SegmentAnything3

model = SegmentAnything3(model_id="sam3/sam3_final")

prompts = [
    # Segment every instance of a concept
    Sam3Prompt(type="text", text="person"),
    # Box one example object and segment every similar instance.
    # box_labels: 1 = positive exemplar, 0 = negative exemplar.
    Sam3Prompt(
        type="visual",
        boxes=[Sam3Prompt.Box(x=1409, y=705, width=112, height=183)],
        box_labels=[1],
    ),
]

response = model.segment_image(
    image="path/to/your/image.jpg",
    prompts=prompts,
    output_prob_thresh=0.5,
    format="polygon",  # or "rle", "json"
)

for prompt_result in response.prompt_results:
    print(prompt_result.echo.text, len(prompt_result.predictions), "instances")

Weights download automatically on first use.

Interactive segmentation in Python

Sam3ForInteractiveImageSegmentation implements the SAM2-style point and box interface, for human-in-the-loop mask refinement:

from inference.models.sam3 import Sam3ForInteractiveImageSegmentation

model = Sam3ForInteractiveImageSegmentation(model_id="sam3/sam3_final")

embedding, img_shape, image_id = model.embed_image(image="path/to/image.jpg")

masks, scores, logits = model.segment_image(
    image_id=image_id,
    prompts={"points": [{"x": 500, "y": 400, "positive": True}]},
)

Use SAM3 in Workflows

Two SAM3 image blocks are available in Workflows:

• SAM 3 runs concept segmentation. Enter the classes you want in class_names (for example ["person", "vehicle"]) and the block outputs instance segmentation predictions that other steps can consume.
• SAM 3 Interactive runs promptable visual segmentation. Supply labeled points (kind labeled_points), for example [{"x": 320, "y": 240, "positive": true}], and optionally connect detections from another model to the boxes field. Each box becomes a separate prompt, and its class name is forwarded to the predicted mask.

Video tracking

The SAM3 Video Tracker block (roboflow_core/sam3_video@v1) runs SAM3's streaming concept tracker frame by frame. You provide concepts as text in class_names, and the model runs fused detection and tracking on every frame. Objects matching a concept keep a stable tracker_id, and, unlike detector-seeded tracking, objects that enter the scene mid-stream are picked up automatically with no re-prompting and no upstream detection model. Each mask carries the concept it matched as its class name and the model's detection score as its confidence (filter with threshold, default 0.5).

• Stateful and local-only. One tracking session is kept per video_metadata.video_identifier. The block requires WORKFLOWS_STEP_EXECUTION_MODE=local, a GPU, and a persistent WebRTC session.
• No prompt scheduling. Concept prompts are registered once per session; the session is re-seeded only when the stream restarts or class_names changes. For detector-driven (box-prompted) video tracking, use the SAM2 Video Tracker block on the SAM2 page, which also accepts sam3trackervideo as model_id.
• Model. model_id defaults to sam3video, the HuggingFace transformers port of SAM3 video, which exposes the frame-by-frame streaming interface. The native sam3 package's video predictor requires the whole video upfront and cannot be used for live streams.

from inference_sdk import InferenceHTTPClient
from inference_sdk.webrtc import StreamConfig, VideoFileSource

WORKFLOW = {
    "version": "1.0",
    "inputs": [{"type": "InferenceImage", "name": "image"}],
    "steps": [
        {
            "type": "roboflow_core/sam3_video@v1",
            "name": "tracker",
            "images": "$inputs.image",
            "class_names": ["person", "forklift"],
            "threshold": 0.5,
        },
    ],
    "outputs": [
        {
            "type": "JsonField",
            "name": "predictions",
            "selector": "$steps.tracker.predictions",
        }
    ],
}

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="YOUR_API_KEY",
)

session = client.webrtc.stream(
    source=VideoFileSource("path/to/video.mp4"),
    workflow=WORKFLOW,
    config=StreamConfig(data_output=["predictions"]),
)

@session.on_data("predictions")
def handle_predictions(predictions, metadata):
    print(predictions)

session.run()

SAM3-3D (beta)

SAM3-3D turns a 2D image plus masks into 3D assets: meshes and Gaussian splats.

Install the dependencies (Python 3.10 recommended):

pip install --no-cache-dir --no-build-isolation -r requirements/requirements.sam3_3d.txt

Or build and run the 3D-enabled GPU container:

docker build -t roboflow/roboflow-inference-server-gpu:dev -f docker/dockerfiles/Dockerfile.onnx.gpu.3d .
docker run --gpus all -p 9001:9001 roboflow/roboflow-inference-server-gpu:dev

Input. An RGB image plus mask_input, which defines the object regions. Masks are accepted as binary arrays ((H, W) or (N, H, W)), COCO flat polygons, point-pair polygons, RLE dicts, or an sv.Detections object from SAM2 or another segmentation model.

Output. mesh_glb (combined scene mesh, GLB), gaussian_ply (combined Gaussian splat, PLY), objects (per-object mesh_glb, gaussian_ply, and metadata with rotation, translation, and scale), and time.

import os

os.environ["SAM3_3D_OBJECTS_ENABLED"] = "true"
os.environ["SPARSE_ATTN_BACKEND"] = "flash_attn"
os.environ["ATTN_BACKEND"] = "flash_attn"

from inference import get_model
from inference.core.entities.requests.sam3_3d import Sam3_3D_Objects_InferenceRequest

model = get_model("sam3-3d-objects", api_key="YOUR_API_KEY")

request = Sam3_3D_Objects_InferenceRequest(
    image={"type": "file", "value": "image.jpg"},
    mask_input=mask_polygons,  # polygons, binary masks, or RLE
)

response = model.infer_from_request(request)

if response.mesh_glb is not None:
    with open("out_mesh.glb", "wb") as f:
        f.write(response.mesh_glb)

for index, obj in enumerate(response.objects):
    if obj.gaussian_ply is not None:
        with open(f"out_object_{index}.ply", "wb") as f:
            f.write(obj.gaussian_ply)

Setting SPARSE_ATTN_BACKEND and ATTN_BACKEND to flash_attn speeds up the pipeline. In Workflows, SAM3-3D supports local execution and remote execution through the sam3_3d_infer() client method or the /sam3_3d/infer endpoint.

See also

• SAM2 - point and box prompted segmentation, plus detector-seeded video tracking.
• Segment Anything (SAM) - the original single-object model.