Datasources

Datasources let you continuously mirror images and metadata from cloud storage into your Roboflow asset library. Once mirrored, images are searchable by semantics, custom metadata, tags, or image similarity, and can be added to any Project for labeling and training.

Currently, AWS S3, S3-compatible storage, Google Cloud Storage, and Azure Blob Storage bucket mirroring is supported.

How Bucket Mirror Works

When you configure a Datasource, Roboflow crawls your bucket and imports all matching image files into your Workspace's Asset Library.

• Supported image formats: JPEG, PNG, BMP, WebP, AVIF
• Files already present in your workspace (matched by their bucket location and hash) are not re-imported, reducing egress costs
• If a .json sidecar file exists alongside an image with the same base name, its metadata is imported; nested keys are flattened using dot notation (e.g., capture.temperature) - see Metadata Sidecars
• Files that disappear from the bucket are kept by default; enable orphan removal to delete them instead (see Removing Orphaned Files)

Mirror your Bucket To Roboflow

Prerequisites

1. A cloud storage bucket (AWS S3, Google Cloud Storage, or Azure Blob Storage) containing your image data
2. A reusable Roboflow credential that can read from that bucket. See Datasource Credentials, then follow AWS S3 Credentials or Google Cloud Storage Credentials.

Add a Credential in Roboflow

Roboflow stores your bucket access securely and encrypted as a reusable credential. For provider-specific setup steps and least-privilege guidance, use AWS S3 Credentials or Google Cloud Storage Credentials.

Go to Credentials in your Workspace settings and click on Add Credential.

Configure a Datasource for Bucket Mirroring

Create a new Datasource from your workspace settings. The form has two tabs:

• "Connection" holds the bucket details and access: name, provider, bucket, region, and Credential. Select your saved Credential from the "Credential" dropdown, or use the "+" next to it to add one without leaving the form.
• "Mirror Configuration" holds the import destination, file filters, and mirror behavior.

Choosing an Import Destination

Under "Mirror Configuration", the "Import Destination" section controls where mirrored files land. Each Datasource imports to a single destination; add another Datasource to import elsewhere.

• "Workspace" mirrors into the Asset Library. Use the "Import into" dropdown to keep files at the workspace root, or select a Project to also add them to that Project.
• "Folder" scopes the mirrored images to a project folder so only that folder's team can see them. This option is available on plans with Project Folder Permissions.

Filtering with Glob Patterns

By default, all supported image files in the bucket are imported. You can restrict which files are imported using glob patterns, either specified directly or via a .txt file stored in the bucket.

You may also provide an explicit whitelist of file paths instead of glob patterns.

Pattern semantics

• * matches any characters except / (single directory level)
• ** matches any characters including / (multiple directory levels)

Examples

Match by prefix:

harvest**

Matches: harvest, harvest2024, harvest/sun/file.jpg, harvest-data.png
Does not match: Harvest, my-harvest

Match everything in a folder:

/harvest/sun/**

Matches: /harvest/sun/file.txt, /harvest/sun/subfolder/image.jpg, /harvest/sun/deep/nested/path/data.png
Does not match: /harvest/moon/file.txt, /other/sun/file.txt

Match by suffix within a subtree:

/planting/**/*crops.png

Matches: /planting/wheat-crops.png, /planting/subfolder/rice-crops.png
Does not match: /planting/wheat.png, /other/wheat-crops.png

Match at a specific directory level with a name pattern:

/*/a/**/*weed*2025-10-27.png

Matches: /farm/a/field/weed-2025-10-27.png, /garden/a/plot/seaweed-data-2025-10-27.png
Does not match: /farm/b/field/weed-2025-10-27.png

Exact path:

/exact/path/to/file.jpg

Matches only that specific file.

Literal wildcards in filenames:
Wrap the pattern in quotes to treat * as a literal character:

"/path/to/file*.jpg"

Removing Orphaned Files

Orphan removal is off by default, so files that disappear from your bucket are retained. When removeOrphanedSourcesWhenDisappeared is enabled, files that are no longer present in your bucket (or no longer matched by your glob patterns) are removed from your Roboflow workspace, provided they are not referenced by any Project or another Datasource configuration.

This also applies when you delete a Datasource. If orphan removal is enabled and the bucket has been mirrored at least once, images originating from that bucket that are not used by any other Project may be removed by the cleanup worker. The delete confirmation dialog will warn you about this and require explicit acknowledgement before proceeding. To avoid this, disable orphan removal on the Datasource's mirror configs before deleting it.

File Naming

The namingStrategy setting controls how imported files are named and displayed in Roboflow:

Image Updates

When an image in S3 is modified, Roboflow can update the copy in your workspace:

• updateImageWhenNewer (default: true) - re-imports the image when the S3 object is newer than the stored version
• updateImageStrategy - controls how the update is applied; currently overwrite (replaces the existing image) is supported

Metadata Sidecars

Attach metadata to images by placing a .json sidecar file in the bucket alongside each image, using the same base name:

my-bucket/
  images/
    photo_001.jpg
    photo_001.json      # metadata for photo_001.jpg
    photo_002.jpg
    photo_002.json      # metadata for photo_002.jpg

The sidecar file contains key-value pairs:

{
  "camera_id": "cam001",
  "location": "warehouse-3",
  "capture": { "temperature": 72.5, "humidity": 45 }
}

Nested objects are flattened using dot notation. The example above produces:

Sidecar file constraints:

• Maximum file size: 256 KB
• Must be valid JSON
• null and undefined values are filtered out

Metadata Sync Strategies

When an image's metadata sidecar .json file is updated in S3, two settings control how the update is applied:

• updateMetadataWhenNewer (default: true) - re-syncs metadata when the sidecar file is newer than the stored version
• updateMetadataStrategy - controls how the synced metadata interacts with metadata you have set manually via the UI or API:

Triggering a Mirroring

You can trigger a mirror manually at any time from the Datasources list by clicking the play button next to a Datasource.

A manual trigger is subject to the following guards:

• In-progress: If a sync is already running, you cannot start another one until it finishes.
• Cooldown: After a sync completes, manual re-triggers are blocked for 15 minutes. The button tooltip shows how many minutes remain. The cooldown is skipped when:
  • The previous sync found nothing new to import (zero files enqueued, or all files failed).
  • You edited the Datasource configuration since the last run.
  • The last run completed with errors.
• Hourly cap: A Datasource can be synced at most 10 times per rolling hour. This cap applies even when a cooldown skip would otherwise allow an immediate retrigger.

Scheduled (cron) syncs bypass both the cooldown and the hourly cap.

Running on a Daily Schedule

To mirror automatically, open the "Scheduling" section under the "Mirror Configuration" tab and check "Run automatically on a daily schedule". The datasource then syncs every 24 hours. Scheduling is off by default.

Viewing Synced Assets

Each of your datasource entries has an eye icon that opens the Asset Library filtered to images and videos from that specific Datasource. The icon is disabled until the Datasource has completed at least one sync.

To view all images synced from any Datasource, click "View Datasource Assets" at the bottom of the Datasources list. This link appears once at least one Datasource has run.

Both links navigate to the Asset Library with a pre-filled tag filter so you can browse, search, and manage only the bucket-mirrored subset of your Workspace images.

S3-compatible storage

Datasources work with S3-compatible storage providers that implement the required S3 API operations.

To configure one of these providers:

1. In the Datasource "Connection" tab, select S3 as the provider.
2. Enter the bucket name and credentials as usual.
3. Define the provider's custom endpoint URL.
4. Set the region to auto or the provider-specific region value.

Use the provider's S3 API endpoint for endpoint. Do not use a CDN URL, a public bucket URL, or a browser download URL.

The same glob pattern filtering, metadata sidecar behavior, and mirror settings also work with these providers.

Path-Style Addressing

Some providers expect the bucket name in the URL path (endpoint/bucket/key) rather than in the hostname (bucket.endpoint/key). Once you enter an endpoint, a "Use path-style addressing" checkbox appears in the "Connection" tab. Roboflow preselects the setting that matches your endpoint and keeps your choice if you change it.

Turn it on for MinIO and other self-hosted or NAS gateways, Oracle Cloud compatibility endpoints, any endpoint reached by an IP address or a port, and buckets with a dot in the name. Leave it off for providers that serve virtual-hosted URLs (ex: Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, Alibaba Cloud OSS). If the wrong setting is used, the connection test fails when it lists files, and the error tells you to switch.

Supported S3-compatible storage providers include:

MCP Server

Connect your AI agent to the MCP Server and it can mirror a bucket into a project with these tools: