Rewards

A reward is created when you send an answer to POST /v1/rewards, and read with GET /v1/rewards/{id}:

{
  "object": "reward",
  "id": "rw_00d1h3x3y8evb35vv5bvvbxe2q4g3rrxg6jr",
  "status": "scored",
  "episode_id": null,
  "task": {"id": "aicme/example-001", "version": "tv_3f9a1c07b2e45d18"},
  "judge_version": "j3",
  "scores": {"reward": 0.75, "partial": 0.75, "must_pass": 1.0, "all_pass": 0.0},
  "flags": [],
  "failure": null,
  "client_ref": null,
  "created_at": "2026-10-28T21:31:32.440Z",
  "scored_at": "2026-10-28T21:31:52.940Z"
}

Status

Status	Meaning
`pending`	Still being scored. Returned with `202` and a `Retry-After` header. Poll again
`scored`	Final. `scores` holds the reward
`failed`	Final. We couldn’t score the answer. See A failed reward

The scores

Field	Range	What it is
`reward`	0 to 1	The reward for the answer
`partial`	0 to 1	Partial credit: a denser signal than `reward`
`must_pass`	0 or 1	Whether the answer met everything it must
`all_pass`	0 or 1	Whether the answer met everything

Use reward to compare with AICME’s published results. partial is the denser signal, and must_pass and all_pass are binary rewards, if your trainer wants one.

The flags

Each flag explains a reward of 0. All four scores are 0 when one is set.

Flag	Meaning
`empty_answer`	The answer was empty, or only whitespace
`over_length`	The answer was longer than the task’s soft cap (Limits)
`instructs_judge`	The answer addressed the scorer instead of answering the task
`unscorable`	The answer couldn’t be scored, after repeated attempts

New flags may be added. Treat a flag you don’t know as a reward of 0 with a reason.

A failed reward

status: failed means we couldn’t score the answer. It is never caused by the answer, and it is never a 0:

{"status": "failed", "scores": null, "failure": {"reason": "infra", "retryable": true}}

We retry a reward ourselves before we report it as failed. To try again, send the same answer: a failed reward isn’t reused, so that creates a new one. reason is judge_error or infra, and new reasons may be added.

Versions

Send the version you downloaded and ran, so the answer is scored against that version. The package’s verifier always does.
Leave out version and judge_version to use the ones your lab is pinned to. Your first reward on a task pins the versions current then.
Name a version to score against it. It doesn’t move your pin.

Sending the same answer again

The same answer, sent again for the same task version and judge_version, returns the first reward. It isn’t scored twice. client_ref is optional: your own label of up to 256 characters, returned on the reward.

Waiting instead of polling

Send Prefer: wait=<seconds>, up to 120, to have us hold the request until the reward is final. See Limits.

Get started

Concepts

Status

The scores

The flags

A failed reward

Versions

Sending the same answer again

Waiting instead of polling

​Status

​The scores

​The flags

​A failed reward

​Versions

​Sending the same answer again

​Waiting instead of polling

Status

The scores

The flags

A failed reward

Versions

Sending the same answer again

Waiting instead of polling