# Openhagen Worlds: development-split sample A small, frozen sample of instances from two families of Openhagen Worlds. It shows what an episode consists of: the observation an agent receives, including the complete published scoring rule; the hidden truth the commit is scored against; a legal gold commit; and one complete episode played by the scripted oracle for each family. The sample comes from the development split only. It contains no evaluation instance and no instance of a held-out family, and it is far too small to serve as training data. ## Families | family | what the agent recovers | experiments | records | |---|---|---|---| | trace debugging | the buggy line of a short program and its fix | runs and line traces | 24 | | causal intervention | the direct causes of a target, or the target's mean under a joint intervention, under hidden confounding | single-variable interventions | 24 | Each family contributes eight instances at each of its three calibrated difficulty levels (0, 1 and 2): 48 instance records and 2 oracle episodes. ## Files - `instances.jsonl`: one JSON record per instance. - `oracle_episodes.jsonl`: one scripted-oracle episode per family, on its first level-0 instance (the same instance appears in `instances.jsonl`). - `MANIFEST.json`: record counts, SHA-256 digests of both files, the seeds, and the disjointness counts described below. ### Instance records | field | content | |---|---| | `family`, `level`, `split`, `seed` | the instance's coordinates; `split` is always `dev` | | `instance_hash` | SHA-256 of the canonical static observation; equal hashes mean equal instances | | `observation` | the public observation at the start of the episode, as the agent receives it: the system description, the objective with its complete success rule, the action grammar, the experiment budget and any supplied history | | `turn_limits` | the number of lab-action turns and of sandboxed Python runs allowed (0 where the family has no Python tool) | | `hidden_truth` | what the commit is scored against (below) | | `gold_commit` | a legal commit that the verifier scores as a success at the start of the episode | | `gold_commit_verified` | `true` for every record: each gold commit was scored by the verifier | | `truth_sha256` | SHA-256 of the instance's full hidden truth (canonical JSON), the identity used in the disjointness check | `hidden_truth` holds the part of the hidden instance that defines the answer. - **Trace debugging:** the buggy line, the kind of edit (operator, constant or variable), the line as shown and the intended expression. The intended program returns the example outputs given in the observation, and the success rule accepts any fix that is equivalent to the intended one on the whole input domain. - **Causal intervention:** the question, the direct causes of every observed variable (the hidden causal graph), the observed variables driven by the unobserved common cause, and the answer: the exact set of direct causes, or the target's mean under the joint intervention (rounded to three decimals; the public tolerance in the observation applies). The numerical parameters of the hidden structural model are not included. ### Oracle episodes `turns` lists each action of the scripted oracle with the result the environment returned; `termination`, `reward` and `success` give the outcome. The oracle reads only the public observation. ### Protocol note The success rule in each observation states that a malformed action ends the episode, which is the behaviour of the strict protocol. Under the standard protocol, the prompt adds a note that replaces this wording: a reply without a legal action is not executed, its reason is returned, it costs one lab-action turn, and two such replies in a row end the episode. Experiments outside the stated limits or beyond the budget end the episode under both protocols. The observations themselves are identical under both protocols. ## How the sample was drawn and checked The instances were generated once, on CPU, from seeds 900000 to 900007 of the development split at each calibrated level; no seed was skipped. Every instance is a pure function of its family, split, level, seed and split key, and the split name and the key both enter the instance's seed. The development key differs from the training keys and from the evaluation keys of the reported results. The test key is private and never distributed; because the split name also enters the seed, no development instance can share coordinates with a test instance. Each sampled instance's hidden truth was also compared, by SHA-256 of its canonical JSON, with 6,464 hidden truths per family and level from the suite's other instance sets: 6,000 training instances; 300 development and 100 final-evaluation coordinates of the reported results (supersets of the blocks those results used); and 64 instances of a further certified evaluation set. That is 19,392 comparisons per family and 38,784 in all, with no collision; no comparison set contained a repeated truth either. ## Licence CC BY 4.0 for the sample data. ## Citation ```bibtex @misc{lorup2026openhagenworlds, title = {Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents}, author = {Lorup, Alexander Boesgaard}, year = {2026}, publisher = {Openhagen}, note = {Preprint} } ```