Introduction

The AICME API scores your model’s answers to AICME’s tasks. You download a task, run it with your agent in your own harness, and send us the final answer. We keep the rubric, the reference solution and the judge, and return a reward.

Base URL

All endpoints are served from a single base URL:

https://api.aicmelabs.com

Authentication

Every request is authenticated with your API key, sent as a bearer token in the Authorization header:

curl https://api.aicmelabs.com/v1/rewards/REWARD_ID \
  -H "Authorization: Bearer your-api-key"

AICME issues your lab’s key when you join. A missing, unknown or revoked key returns 401 unauthenticated. See Authentication for the full model.

Errors

Errors share one envelope, error, with a machine-readable code and a human-readable message:

{
  "error": {
    "code": "not_found",
    "message": "No such task version, or none this key can use.",
    "request_id": "req_00d1ghp0tcar9fcc8pa11p3jp4jt7wnb71eh"
  }
}

Status	Code	Meaning
400	`invalid_request`	The request body failed validation.
401	`unauthenticated`	The API key is missing, unknown or revoked.
403	`forbidden`	The key is valid, but can’t use this route.
404	`not_found`	The resource doesn’t exist, or your lab can’t see it.
410	`version_retired`	The task version is past its sunset. Move to the current one.
413	`request_too_large`	The body is over 512 KB.
500	`internal`	Our fault. Send us the `request_id`.
503	`service_unavailable`	Retry after the `Retry-After` header.

See Errors for every field and code.

Typical flow

Scoring runs asynchronously. Download a task, send an answer, then poll for the reward until it’s ready.

Download a task

Call GET /v1/tasks/{task_id}/versions/{version}/package. You receive the task as a Harbor package, a plain tar, to run with your agent.

curl https://api.aicmelabs.com/v1/tasks/aicme/example-001/versions/tv_3f9a1c07b2e45d18/package \
  -H "Authorization: Bearer your-api-key" \
  -o package.tar

Send an answer

Call POST /v1/rewards with the task, the version you ran, and your agent’s final answer. You receive the reward and its id, usually still pending, with 202 Accepted. Inside Harbor, the package’s verifier makes this call for you.

curl -X POST https://api.aicmelabs.com/v1/rewards \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"task": {"id": "aicme/example-001", "version": "tv_3f9a1c07b2e45d18"}, "answer": "Your final answer here"}'

Poll for the reward

Call GET /v1/rewards/{id} while status is pending, at least 5 seconds apart. It ends as scored or failed.

curl https://api.aicmelabs.com/v1/rewards/REWARD_ID \
  -H "Authorization: Bearer your-api-key"

Read the reward

On success, scores.reward is your reward, between 0 and 1. See Rewards for the other scores.

{
  "id": "rw_00d1h3x3y8evb35vv5bvvbxe2q4g3rrxg6jr",
  "status": "scored",
  "scores": {"reward": 0.75, "partial": 0.75, "must_pass": 1.0, "all_pass": 0.0},
  "flags": []
}

Learn more

Quickstart

From a key to your first reward.

Authentication

Keys, and what a key can see.

Errors

Every code, and which to retry.

Limits

How long an answer can be, and how long to wait.

Get started

Concepts

Introduction

Base URL

Authentication

Errors

Typical flow

Learn more

Quickstart

Authentication

Errors

Limits

​Base URL

​Authentication

​Errors

​Typical flow

​Learn more

Quickstart

Authentication

Errors

Limits

Base URL

Authentication

Errors

Typical flow

Learn more