CoreLabel Documentation

Platform overview

CoreLabel Datum is an async data annotation pipeline. You upload a dataset, the platform pre-labels it with AI, routes low-confidence items to human reviewers, runs quality checks, and delivers a structured export.

The pipeline runs entirely in the background — submit a job and come back when it's done. You never label in Datum itself: human annotators work in Label Studio (files and agent trajectories) or Robotix (robot recordings), and Datum is the orchestration layer between your raw data and your annotation output.

RL-trajectory projects — robot recordings and agent runs — skip the file pipeline and become episodes instead. See Robotics projects and Agent projects.

Quickstart

Projects

A Project groups jobs that share the same data type and label schema. Create one project per annotation task — for example, one for "product images" and a separate one for "customer reviews".

Data types

You cannot change the data type after a project is created. If you need a different type, create a new project.

Uploading data

Go to Upload and drag files into the drop zone, or click to browse. Files upload directly to secure object storage — the backend never proxies your bytes, so large files are fast.

File-type validation

The allowed extensions depend on the data type you select. Rejected files show a ✗ icon and are skipped.

Progress tracking

Each file shows its own status icon: ⏳ uploading → ✓ done or ✗ failed. The overall progress bar updates as files complete. When all files finish, you're automatically redirected to New Job with the storage prefix pre-filled.

Robotics and agent projects are deliberately absent from this page's project list. Both become episodes rather than job input, so they are added from the project instead — see Robotics projects and Agent projects.

Cloud connector

Instead of uploading files, you can link an existing cloud bucket. CoreLabel reads data in-place — your files never leave your storage. Only annotation outputs are written back to CoreLabel's storage.

AWS S3

1. Switch to the Cloud Connector tab on the Upload page.
2. Select AWS S3 and enter your bucket name.
3. Enter an Access Key ID and Secret Access Key for an IAM user with s3:ListBucket and s3:GetObject permissions on that bucket.
4. Click Verify connection and wait for the status to turn green.

Google Cloud Storage

1. Select Google Cloud Storage and enter your bucket name.
2. Paste the full JSON contents of a service account key file. The service account needs the Storage Object Viewer IAM role.
3. Click Verify connection.

Verification

Verification lists your bucket to confirm access and counts the files. This typically takes 5–30 seconds. The connector must be in verified status before you can create a job with it.

Jobs & pipeline

A Job is one annotation run. It takes a dataset and a project, then moves through the following pipeline stages automatically.

Pipeline stages

Human review routing

Items where the AI confidence score falls below your project’s threshold (default 0.75) are escalated for human review. This happens on every plan — a plan decides how much AI pre-labeling you get, never whether an uncertain item is allowed to be seen.

Escalated items are never discarded. They are either sold to you as a lot and routed to annotators, or the job finishes with the AI label and the item is reported as unreviewed. Which of those happened is visible per item on Prelabel review.

IAA score

Inter-Annotator Agreement (IAA) measures how consistently two raters labelled the same items — the AI pre-label against a human annotation, or a human against the senior reviewer who re-checked their work. It ranges from 0.0 (no agreement) to 1.0 (perfect agreement).

It is Cohen's κ for classification and mean IoU for detection; the job tells you which, because the two are not comparable to each other. Where tasks went to a senior reviewer as well, a three-rater Fleiss' κ is also recorded.

If IAA is consistently low, consider adding more examples to your label guidelines or reducing the number of label classes.

Robotics projects

A project whose data type is RL trajectory and whose pillar is robot works differently from every other project, because its data is a recording rather than a pile of files. It has no Upload step, no jobs, and no AI pre-labeling — a recording becomes an episode, and the episode is what gets annotated.

Adding data

Open the project and use + Add data. Everything arrives through the one page:

Formats

The format is detected, not declared

You are not asked to pick one. The file extension does not settle it — a .zip carries a LeRobot dataset and a ROS 2 bag directory alike — so the contents are read after the upload lands, and the result appears in the imports table marked detected.

An archive matching two formats is refused naming both rather than resolved by picking one. Guessing wrong would not produce a failed import; it would produce a successful one holding the wrong data under a name that looks right. If you know better than the probe, set the format explicitly under Advanced.

Video + table: name the clock

A table of numbers does not say what its timestamps mean. 1.5, 1500 and 1757000000 are all plausible readings of the same instant — seconds, milliseconds, or a Unix epoch — so you are asked which column holds the time and what unit it is in, under Uploading a video with a CSV? on the Add data page.

Rows must be in time order, and text columns are dropped rather than converted — a coerced text column becomes numbers an annotator reads as sensor dropout.

RLDS: give the control rate

RLDS stores the order of an episode's steps and nothing about their timing — no timestamps, no frame rate. So the signals and the camera frames have no clock until you supply one: under Uploading an RLDS dataset? on the Add data page, give the rate the robot was controlled at (DROID: 15 Hz). Camera frames are encoded into a video at that rate, frame for frame.

Steps' boolean flags (is_first, is_last) and very wide features such as a 512-number language embedding are not imported as signals. They are listed on the episode rather than dropped silently. The episode's language instruction becomes its task.

Several recordings in one upload

An upload holding several bags or MCAP files imports the one at the episode or recording index you give (counting from 0), and the imports table shows which of how many it read. Upload the others separately, or re-import with a different index.

Define your objects first

Only terms of kind object become labels an annotator can draw. Materials and affordances are vocabulary: a box drawn around “graspable” would record a property as though it were a physical thing.

Why there are no jobs

Jobs exist to run AI pre-labeling over individual files and route the uncertain ones to human review. That pipeline cannot read a recording, so robotics projects do not offer jobs at all — there is no quota consumed, and no purchase gate. Human annotation happens in Robotix, below.

Agent projects

An agent project holds trajectories: recorded runs of an AI agent, one tool call per step. Each becomes an episode you can replay step by step, score, and send to human annotators who judge every step. Create one with data type RL trajectory and kind Agent — the kind defaults to Robot, so choose it deliberately.

Adding trajectories

Use + Add trajectories on the project page. Upload and New job do not list agent projects: their pipeline would read a trace as plain text and it would never become an episode. The page takes a .jsonl file (one trajectory per line) or a .json array — including conversations logged straight from the OpenAI or Anthropic APIs, OpenTelemetry exports, and the files agent harnesses leave behind: τ-bench and τ²-bench results, SWE-agent .traj, Inspect AI .eval logs, OpenHands event streams, and OpenAI Agents SDK trace exports. A harness’s own verdict (reward, score, expected actions) is kept as a hidden answer key and never shown to annotators. A bad line is listed with its reason and does not stop the rest.

Tick Score each trajectory to run the four scoring reports: tool & API schema, multi-agent handoffs, long-horizon memory, and efficiency. A report that has nothing to measure shows a dash, never a zero.

The file formats

One .jsonl line per trajectory, or a .json array. Any of them may be zstd-compressed (.zst) — the server decompresses it — and a single file may mix all of these shapes. Each line is recognised by what it contains, not by a name you give it.

A line that matches none of these is refused on its own, with the reason and the line number, while the rest of the file still loads. A conversation carrying both OpenAI and Anthropic markers is refused too, rather than read half wrong.

The otel name under framework trace reads a simplified span list, not a real OpenTelemetry export — for those, upload the OTLP JSON itself.

What an annotator sees

Each step is shown as one turn: who acted, then what they were answering and thinking, then the call and what came back. Whatever format you upload, these are the fields it is read from — fill them in pre-shaped steps to get the most out of labeling:

AutoGPT’s thoughts and a node graph’s thought are shown the same way. The AI flagger reads exactly what the annotator reads, so on traces with reasoning it sees that too — its measured numbers come from traces without reasoning.

Node graphs in more detail

Each terminal path becomes its own episode — one coherent run an annotator can read top to bottom. Flattening the whole tree into one list instead would interleave branches, and a retry from a different branch would read as the agent repeating itself. A node on two paths therefore appears in both episodes, and the path id is part of the episode's reference.

What else comes across:

• task.prompt becomes the goal on every step, and oracle_steps the efficiency baseline.
• state_delta becomes what a step established. Nothing invents what a later step needed, so memory recall stays unmeasured rather than guessed.
• Per-node annotation labels and violations are kept as the episode's answer key: scored against later, never shown to an annotator, and never translated into Datum's own labels. Their words are not our words, and the overlap is a guess.
• No tool schema is built from allowed_tools: it lists names without parameters, and the scorer compares parameter keys exactly — so a schema built from it would mark almost every real call wrong.

Sending for labeling

Send N for labeling — on the project’s episode list or under the Add trajectories results — sends them as one Label Studio project, one task per trajectory, so an annotator works a single queue. Trajectories already sent are left out. A whole trajectory is always one task, never one task per step, because a wasted retry only looks wasteful beside the step it repeats.

Step labels

Links between steps record a handoff to another agent, a delegation, or a step that recalls a fact established earlier.

AI step-flagging (optional)

Tick Flag suspicious steps with AI first when sending, and a model marks the steps worth a look before an annotator opens them:

• It marks steps needs review with a note saying why, and deliberately never names the fault. Measured against known-flawed traces, 93% of the steps it flagged really were flawed — but it often named the wrong fault, so the label stays the annotator’s.
• It finds under half of the flaws. It tells an annotator where to look first; it is never a reason to skip reading a step.
• Where a trajectory has a tool & API schema report with a ground-truth schema, wrong tool and wrong params are labelled exactly instead, because that comparison cannot be wrong.
• Handoff, delegation and recall links are drawn as well. They need no ground truth.
• Everything arrives as a suggestion, never a saved label. Unflagged steps carry no mark, nothing is pre-filled as correct, and no confidence is filled in.
• The most-flagged trajectories are placed first in the annotator’s queue.
• It costs one pre-label unit per trajectory. If your allowance is used up, the trajectories are still sent — just without AI flags.

Checking your annotators

A golden task can be a whole trajectory with known per-step labels. It is mixed invisibly into labeling batches and scored when it comes back, which gives each annotator a skill profile. It is scored only on the steps the reference names; a named step left blank counts against the annotator. A golden is never saved as a real label on the episode it came from.

Robotix

Robot and sensor data is annotated in Robotix, a separate workspace built on CVAT. Image and text jobs stay in the standard pipeline; robotics episodes go to Robotix because they need tools the general annotation view does not have — 3D point clouds, synchronised video, and force traces sampled far faster than any camera.

Getting there

Open an episode from its project; its annotation tasks are listed under Annotate in Robotix, and each opens in a new tab.

You are signed in automatically. Open creates your studio account the first time, places you in your organization’s workspace and signs you in with your CoreLabel account — there is no separate studio password. Each Open uses a link that works once, for one minute, so if the studio ever shows its login page, go back and press Open again.

Reading the task list

Two markings on a row change what you should do with it.

Two kinds of task

An episode produces one task per sensor, and the tools you get depend on what that sensor records.

Object names in a shapes task come from your project’s ontology. Add a term there and it appears as a label; only physical objects are drawable, so categories like “graspable” stay as properties rather than things you can box.

Interval labels

Interval tasks share one vocabulary, so a span means the same thing whoever recorded it.

Mark each span with how sure you are — certain, probable or unsure. That rating is stored with the annotation and used when the data is scored, so an honest “unsure” is worth more than a confident guess.

Suggested spans

Some intervals are already drawn when you open a task. These are automatic suggestions, not answers: they are detected from the force trace and stay marked as suggestions until you accept them. Nothing counts as labelled until a person confirms it. Adjust the edges, correct the label, or delete a span that is wrong — the detector is right roughly twice as often as chance, which makes it a useful starting point and not something to trust unchecked.

Reading a force trace

Two things are worth checking before you label anything.

Whether the trace is compensated. The calibration panel shows which calibration produced the numbers on screen. An uncompensated channel can carry a steady offset of around 14 N from the weight of the tool itself — enough to look like sustained contact when the arm is simply holding still.

Where the timing comes from. The timeline draws camera frames as ticks rather than bars, because a 30 Hz camera has thirty instants per second and nothing in between. If a lane is flagged as being on a different clock, treat its alignment with the force trace as approximate.

On a tactile pad, hatched cells are dead taxels — no reading at all. They are not zero, and a contact that appears to stop at a hatched cell has not necessarily stopped.

Video tasks

Boxes drawn on a camera task are saved back. They are recorded against the instant the frame falls on rather than a step number, because a camera task is timed by the video’s own measured frame rate and the episode it belongs to may have no steps at all — a LeRobot import carries video and sensor traces but nothing step-shaped.

One thing camera tasks still do not get is a shape-quality score. That number is the studio’s comparison against a ground-truth job, and a camera task has none; it reads as not measured rather than as a zero.

Saving

Save in the studio as you normally would. Annotations return to CoreLabel automatically and quality is measured on arrival, so there is no separate submit step. Saving repeatedly is safe: each save replaces the previous one for that task rather than adding duplicates.

Quota

Each data item in a job consumes one AI pre-label. So does each agent trajectory sent with AI step-flagging ticked — one unit per trajectory regardless of length, charged only if the model actually runs. Human labeling is bought per batch and is never counted against quota. Usage is tracked per calendar month (UTC), resets on the 1st, and does not roll over.

Check current usage on the Quota page. Quota is reserved upfront when you submit a job; if the job fails it is not automatically refunded.

Plans

A plan is a monthly AI pre-label allowance and nothing else. Every export format, webhook and integration is available on every plan; there is no feature a higher plan unlocks.

Human labeling

A plan buys AI capacity. Human labeling is bought per batch, after the AI has run, so you decide with the confidence numbers in front of you rather than in advance.

When pre-labeling finishes, a self-serve job pauses at Awaiting decision with a quote covering exactly the items that fell below your threshold. Price depends on data type and how complex the work is; complexity is set by CoreLabel, not chosen at checkout.

Your three options

• Accept — those items, and only those, go to human annotators. The job then continues through QA and export.
• Decline — the job completes now with AI labels. Nothing is billed.
• Do nothing — the quote expires and the job completes exactly as if you had declined. A job never sits waiting indefinitely.

A quote freezes its own numbers. If the price list changes between you seeing a quote and accepting it, you pay what you were shown.

Projects run as a managed engagement by CoreLabel skip this step: labeling is covered by the engagement, so there is nothing to approve.

Billing

Go to Billing to upgrade your plan. Payments are processed securely through Stripe — CoreLabel never stores your card details.

After a successful payment your plan updates immediately; you don't need to log out or refresh. Downgrading to Free is immediate and self-service — there is no payment involved in moving to a $0 plan, so it needs no support ticket.

Export formats

When a job completes, click ↓ Data on the Jobs page to download the annotation export in your project's configured format — JSON, CSV, COCO, YOLO, or Pascal VOC. The export contains every item with its labels, confidence scores, annotator metadata, and audit trail.

JSON structure

{
  "job_id": "abc123",
  "project": "Product images",
  "completed_at": "2026-06-28T12:00:00Z",
  "iaa_score": 0.87,
  "items": [
    {
      "id": "item_001",
      "source_key": "uploads/abc123/cat.jpg",
      "labels": [
        { "class": "cat", "confidence": 0.94, "bbox": [10, 20, 200, 180] }
      ],
      "reviewer": "human",
      "annotated_at": "2026-06-28T11:55:00Z"
    }
  ]
}

Other formats

COCO, YOLO, and Pascal VOC export formats are available on Scale and Enterprise plans. Custom schema export is available on Enterprise. Select the format when creating the project.