Inference Server

The Inference Server is a standalone microservice that wraps the inference Python package. It exposes HTTP endpoints for images and a WebRTC endpoint for video streams. One server can serve multiple clients and run the same models and Workflows as Roboflow's hosted APIs. It is the recommended way to self-host: see Pick a path for how it compares to running the library directly.

Where it runs

Self-host the server on your own hardware (Raspberry Pi, NVIDIA GPU, NVIDIA Jetson, or a plain server) with Docker, or in your own AWS, GCP, or Azure account. Roboflow also runs the same server for you as the Serverless Cloud API and Dedicated Deployments: see Choosing a Deployment Option.

Whichever you pick, you talk to it through the Inference SDK, because they share one interface: only the api_url changes.

Running with Docker

Before you begin, make sure Docker is installed on your machine. The easiest way to start the Inference Server is with the Inference CLI:

pip install inference-cli && inference server start

This pulls the appropriate Docker image for your machine, with dependencies pre-installed, and starts the Inference Server on port 9001. Check the server status with:

inference server status

Manually setting up a Docker container

inference server start runs docker run under the hood with recommended security settings, caching, and platform-specific options.

If you want to start the container yourself, see the "Manually starting the container" section of your platform's install guide:

• Linux
• Windows
• Mac
• Jetson
• Raspberry Pi

Container settings are controlled with environment variables: see Docker configuration options and the full environment variable reference.

Built-in JupyterLab notebook

Inference Servers ship with a built-in JupyterLab environment, which is the fastest way to experiment during development and testing. It is disabled by default, so start the server with the --dev flag to enable it:

pip install inference-cli
inference server start --dev

Then open http://localhost:9001 in your browser to see the Inference landing page, which links to resources, examples, and the built-in JupyterLab environment. Select "Jump Into an Inference Enabled Notebook" to open JupyterLab in a new tab. It comes preloaded with example notebooks and all the dependencies needed to run Inference.

Stream video

Use the Inference SDK WebRTC client to stream webcams, camera feeds, and video files through a model or Workflow:

pip install "inference-sdk[webrtc]"

Set api_url="http://localhost:9001" when you create InferenceHTTPClient. See WebRTC Streaming for model and Workflow examples, or follow Video processing with Workflows for a task-based guide.

In this section