API documentation
Hosted evaluation of the Openhagen Worlds final instances
Access tokens are issued on request: email hello@openhagen.ai with your name, affiliation and the models you intend to evaluate.
Email hello@openhagen.aiService: https://eval.openhagen.ai
The service plays any model on the final evaluation instances of the Openhagen Worlds paper, episode by episode over HTTPS: start an instance, send the model's reply to each observation, and receive the next observation, the reward, whether the episode is done and, at the end, whether it succeeded. The hidden truth, score details, scripted oracle, generators, instance coordinates and keys never leave the server.
What is hosted
Exactly the paper's final instances: 24 per family and level, generated once from instance indices disjoint from development, on the family code of the paper's evaluations.
| group | families (API identifier) | levels | instances |
|---|---|---|---|
| discovery worlds, trained | circuit fault (circuit_fault), impedance topology (impedance_topology), heat inclusion (heat_inclusion), hidden law (hidden_law), control plant (control_plant), reaction network (reaction_network), pathway fault (pathway_fault), knockout network (knockout_network), group testing (group_testing), rule induction (rule_induction), causal intervention (causal_intervention), trace debugging (trace_debugging) | 0, 1, 2 | 864 |
| discovery worlds, held out | mechanism discrimination (mechanism_plus), counterfeit coin (counterfeit_coin), automaton learning (automaton_learning) | 0, 1, 2 | 216 |
| MicroWorld | certificate audit (receiptfall), exact execution (exact_execution) | 0-5 | 288 |
| PhysicsWorld | rigid-body identification (rigid_body_id), airfoil inference (aero_redux), optical bench (optics_bench), coupled oscillators (oscillator_network), fluid network (fluid_network, held out) | 0-3 | 480 |
| ChemistryWorld | structure elucidation (structure_elucidation), titration equilibria (titration_equilibria), kinetics process design (kinetics_process_design), chromatography method (chromatography_method), electrochemical mechanism (electrochem_mechanism) | 0, 1, 2 | 360 |
| total | 27 families | 92 levels | 2,208 |
An instance is addressed by family, level and index (0-23); GET /v1/catalog lists every cell. Each observation carries an instance_hash (SHA-256 of the instance's opening public state). The instance-set digest instances_sha256 in /health and /v1/catalog is the SHA-256 of the lines env_id|level|index|instance_hash over all 2,208 instances, sorted and joined by newlines, so a client that plays every instance can recompute it.
The simulators are those of the paper's evaluations: the finite-volume heat solver, MuJoCo 3.16.0 (rigid-body identification) and libRoadRunner 2.10.0 (pathway fault), on Linux x86_64. In /health, vendored_tree_sha256 identifies the family code and platform the machine.
family code (vendored_tree_sha256) | 56d37df576fc16715fd24a2f2302838c5f08f19cdb58f2c7cc549d3f90096eac |
instance set (instances_sha256) | c17904f9fba8cb2f4e5eb30134608e700ee5856309f132305c6effe8adb6da68 |
Protocol
The service applies the paper's standard protocol and caps; the model settings are the client's to apply.
- Observation. The exact chat messages the paper's models received:
systemandprompt(the user message). The prompt re-renders the whole public state every turn (laboratory state, objective with its complete scoring rule, action grammar, experiment budget, readings so far, earlier Python output), so each decision is a fresh two-message conversation[system, prompt]; no conversation memory is needed. - Reply. Passed verbatim to the episode parser, as in the paper's harness; only the text after the last
</think>is read. A legal reply is oneACTION: <command>line in the family's grammar, a JSON object{"action": "<command>"}, or, in families with the Python tool (python_tool: true), exactly one```pythonblock. The block runs in a sandbox with the readings asDATAand the public state asLAB; its output appears in the next prompt. - Standard protocol. A reply without a legal action is not executed: it costs one laboratory turn, and the next prompt gives the parser's reason and the legal form. Two such replies in a row end the episode at zero, as does an experiment outside the safe limits or beyond the budget. A Python run costs no experiment; at most four run per episode.
- Caps. 24 decisions per episode, plus the instance's experiment budget and laboratory-turn limit (both in the observation). An episode ends on a commit, on an ending above, at the turn limit or at 24 decisions.
- Reward. 1 for a correct commit, 0 otherwise;
successisreward > 0. Commits are scored only inside an episode, and only the reward is returned. - Model settings of the paper. Thinking on, at most 4,096 thinking tokens per decision; if the thinking has not closed by then, append
"\n</think>\n\n"and let the model answer. The answer is limited toanswer_token_budgettokens (in every observation). Temperature 1.0, top-p 0.95, top-k 20. Send the answer text (or the full text including</think>) as the reply.
Report per-family success pooled over its levels (24 instances per level), with the protocol, the thinking budget and the invalid-ending rate (termination starting with invalid).
Access
Send the token as Authorization: Bearer <token> on every /v1 call except the catalogue. Tokens are issued on request by Openhagen (state your name, affiliation and the models you intend to evaluate) and can be revoked.
Endpoints
All bodies are JSON.
| method | path | body | returns |
|---|---|---|---|
| GET | /health | liveness, readiness, family-code hash, instance-set digest (no token) | |
| GET | /v1/catalog | hosted families, levels, protocol, model settings and limits (no token) | |
| POST | /v1/episodes | {"env_id": "circuit_fault", "level": 0, "index": 0} | {"episode_id", "observation"} |
| GET | /v1/episodes/{id} | the current observation, or the outcome of a finished episode | |
| POST | /v1/episodes/{id}/step | {"reply": "<model reply>"} | {"observation", "reward", "done", "success", "termination", "kind"} |
| DELETE | /v1/episodes/{id} | abandons a live episode (scored 0) | |
| GET | /v1/results | this token's finished episodes and successes per cell | |
| GET | /v1/usage | this token's limits and current use |
env_id accepts the family identifier or the full environment id (PhysicsWorld/circuit_fault-v1).
Observation fields: episode_id, env_id, family, level, index, system, prompt, lab_state (the public state embedded in the prompt), python_tool, answer_token_budget, thinking_budget, decisions_used, decision_limit, experiments_used, experiment_budget, lab_turns_used, lab_turn_limit, python_runs_used, done, instance_hash. When done is true, success and termination are set (commit, invalid:streak, invalid:turn_cap, unsafe_experiment, budget_exhausted, turn_cap or decision_limit); kind says what the last reply did (experiment, python, commit, invalid).
import requests
URL, H = "https://eval.openhagen.ai", {"Authorization": "Bearer <token>"}
def play(model, env_id, level, index):
r = requests.post(f"{URL}/v1/episodes", headers=H,
json={"env_id": env_id, "level": level, "index": index}).json()
eid, obs = r["episode_id"], r["observation"]
while True:
reply = model([{"role": "system", "content": obs["system"]},
{"role": "user", "content": obs["prompt"]}],
answer_tokens=obs["answer_token_budget"])
out = requests.post(f"{URL}/v1/episodes/{eid}/step", headers=H,
json={"reply": reply}).json()
if out["done"]:
return out["success"], out["termination"]
obs = out["observation"]
Limits
Per token: 240 requests per minute (bursts of 60); 32 live episodes at a time; 3 episodes per instance in total, abandoned and expired ones included, so the test set cannot be searched by repeated attempts; 2,500 episodes per 24 hours (one full pass is 2,208). A reply is at most 65,536 characters. A live episode idle for 6 hours expires and counts as played. A request beyond a limit returns HTTP 429 with the reason (and Retry-After for the rate limit).
The service runs on one always-on CPU machine; after a restart, live episodes resume where they were. Simulator-backed families (fluid network, pathway fault, rigid-body identification) can take tens of seconds per step under load; a full pass is dominated by model inference and takes hours.
What is logged and kept
Request logs hold metadata only: time, route, status, latency, token identifier and a truncated hash of the client address; never replies, observations or tokens. A live episode's replies are stored until it ends, so it can resume after a restart, and are then deleted; the outcome (reward, success, termination, decisions) is kept per token for /v1/results and the limits.
Reproducibility check
Before release, the paper's scripted oracle and naive policy were played against the deployed service on all 2,208 instances. Each policy ran client-side on a local copy of the episode built from the research code, and every reply went to the service over HTTPS.
- Instances and protocol. In 26 of the 27 families (2,136 instances), every observation was byte-identical to the research code's at the same step, for both policies, and every episode outcome agreed.
- Paper numbers. On the fifteen discovery worlds the service reproduces all 30 scripted per-family success rates of the paper's final-instance table exactly (pooled oracle 0.964 on the twelve training families and 0.995 on the three held-out families; naive 0.060 and 0.046). Over all 2,208 instances the oracle succeeds on 2,116 (0.958) and the naive policy on 130 (0.059).
- One platform-dependent family. Pathway fault draws its instances through libRoadRunner, whose floating-point results feed the generator's admission checks, so a few instances differ between processor platforms: 10 of 72 between Linux x86_64 and macOS arm64. The service runs on Linux x86_64, the platform of the paper's evaluations; on its own pathway-fault instances the oracle succeeds on 71 of 72 and the naive policy on 1 of 72, the paper's rates.
/healthreports the platform, andinstances_sha256is bound to it.