Skip to main content

​
The answer’s length

Two limits, with different jobs.

​
The soft cap: a reward of 0

Each task version has a soft cap on the answer, in characters. We send it with the task’s version ID. An answer longer than the soft cap is still a reward: all four scores are 0, with the flag over_length. It is never an error, so a trainer that drops failed requests can’t skip it. Length is counted from the answer we receive, never from anything the client says about it. Keep your agent’s answers under the soft cap. Tell your agent the cap if your prompt allows it.

​
The hard limit: 512 KB per request

A request body over 512 KB gets 413 request_too_large, and nothing is created. This limit guards the transport. It is far above every task’s soft cap, so an answer this long scores 0 anyway. The package’s verifier never sends a 413. It cuts an answer that wouldn’t fit, at a UTF-8 character boundary, and also sends the answer’s original length and SHA-256. The answer still scores 0 with over_length, and nothing past the cap is ever scored. If you post answers yourself, cut them the same way before you send them.

​
Pending rewards: 2,000 per lab

Your lab can have up to 2,000 rewards pending at once. A new answer past that gets 429 rate_limited with a Retry-After header, and nothing is created: post it again once some of your rewards are final. Posting an answer whose reward is pending or scored still returns that reward, and an answer that scores 0 at once, such as an empty one, is never refused. An answer whose reward failed is scored again, as a new pending reward, so it counts.

​
Waiting for a reward

Scoring an answer takes tens of seconds. A reward is final only when its status is scored or failed. A 202 is never final: it carries a reward that is still pending.
  • Ask us to wait. Send Prefer: wait=<seconds> on POST /v1/rewards or GET /v1/rewards/{reward_id}, up to 120 seconds; a larger value is treated as 120. We may hold the request until the reward is scored or failed, and then answer 200. You can still get a 202 with status: pending:
    • at the end of the wait, when scoring took longer;
    • at once, without waiting, when we’re busy or restarting. The response then has no Preference-Applied header.
    When we do wait, Preference-Applied: wait=<seconds> gives the wait limit we applied, not the time we waited.
  • Poll until it’s final. On a 202, read the reward with GET /v1/rewards/{reward_id}, with or without Prefer: wait, until its status is scored or failed. A pending reward comes back with Retry-After: poll no more often than that, and at least 5 seconds apart.
Set your harness’s verifier timeout to allow for a wait; the package’s own setting does. The package’s verifier already polls until the reward is final.