Roboflow Docs

Inference SDK

HTTP client for the Roboflow Inference Server, with local, hosted, async, and batch inference from Python.

The inference-sdk Python package provides InferenceHTTPClient, a client for talking to an Inference Server over HTTP. The same client works against the Roboflow Serverless Hosted API, a Dedicated Deployment, a self-hosted server, or a server running on an edge device - only the api_url changes.

pip install inference-sdk

inference-sdk is a thin HTTP client and does not run models itself. To load and run models inside your own Python process, use the inference package.

Quickstart

You can run inference on images from URLs, file paths, PIL images, and NumPy arrays.

from inference_sdk import InferenceHTTPClient
import os

image_url = "https://media.roboflow.com/inference/soccer.jpg"

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["API_KEY"],
)

results = client.infer(image_url, model_id="soccer-players-5fuqs/1")
print(results)

On the first request against a self-hosted server, the model weights are downloaded and set up. This request may take some time depending on your network connection and the size of the model. Once the model has downloaded, subsequent requests are much faster. You can also pre-load models and manage loaded weights to control this process.

The model ID is composed of the string <project_id>/<version_id>. See Workspace and Project IDs to find yours.

Self-hosted server

You can also self-host the Inference Server (see the Inference CLI), and then change api_url in the InferenceHTTPClient:

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key=os.environ["API_KEY"],
)

AsyncIO client

import asyncio
from inference_sdk import InferenceHTTPClient

CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)

image_url = "https://source.roboflow.com/pwYAXv9BTpqLyFfgQoPZ/u48G0UpWfk8giSw7wrU8/original.jpg"
loop = asyncio.get_event_loop()
result = loop.run_until_complete(
  CLIENT.infer_async(image_url, model_id="soccer-players-5fuqs/1")
)

Parallel and batch inference

You may want to predict against multiple images in a single call. Two parameters of InferenceConfiguration control batching and parallelism:

  • max_concurrent_requests - max number of concurrent requests that can be started
  • max_batch_size - max number of elements that can be injected into a single request

This enables the following improvements:

  • if you run the inference container on a powerful on-prem GPU machine, setting max_batch_size properly may bring throughput benefits
  • if you run inference against the hosted Roboflow API, setting max_concurrent_requests causes multiple images to be served at once, bringing throughput benefits
  • a combination of both options can be beneficial for clients running the inference container on a cluster of machines: the load of a single node can be optimised and parallel requests to different nodes can be made at a time
from inference_sdk import InferenceHTTPClient

image_url = "https://source.roboflow.com/pwYAXv9BTpqLyFfgQoPZ/u48G0UpWfk8giSw7wrU8/original.jpg"

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
predictions = CLIENT.infer([image_url] * 5, model_id="soccer-players-5fuqs/1")

print(predictions)

Methods that support batching and parallelism:

  • infer(...) and infer_async(...)
  • ocr_image(...) and ocr_image_async(...) (enforcing max_batch_size=1)
  • detect_gazes(...) and detect_gazes_async(...) - deprecated, always raises inference_sdk.http.errors.FeatureDeprecatedError
  • get_clip_image_embeddings(...) and get_clip_image_embeddings_async(...)

The client also supports core foundation models (CLIP, DocTR), running Workflows for multi-step pipelines, and WebRTC streaming for real-time video inference. Use WebRTC to process webcams, camera streams, and video files with either a model or a Workflow.

What is actually returned as a prediction?

InferenceHTTPClient returns plain Python dictionaries that are the responses from the model serving API. Modification is done only in the context of the visualization key, which keeps the server-generated prediction visualisation and can be transcoded to the format of choice. Client-side rescaling only adjusts the input size.

Next steps