- Task type
- machine-learning
- Metric
- Official score = 80% normalized area under the best-so-far effective-score curve on a log token axis, plus 20% of its final value
- Score ceiling
- 100
- Human baseline
- unknown (the author notes a task can take a human several hours; no aggregate human score is published)
- Test-set privacy
- public (the author publishes the prepared data file and the full run results; the underlying task datasets are not published)
- Construction
- 11 complex hand-made tasks, each with a novel dataset, a goal the model must infer and feedback limited to what the sandbox reveals; hinted and hintless twins of each task each receive half the task's weight, and effective scores include normalisation and hint penalties, so they are not raw accuracy
- Key assumptions
- Novel hand-built datasets the model has never seen force genuine exploration rather than recall of a familiar pipeline
- Limitations
- Scoring is a weighted curve area rather than a plain accuracy, so the number is not comparable with a raw score; 11 tasks is a small set, and run-to-run variance is wide enough that the author publishes uncertainty intervals
- Contamination history
- Tasks are hand-made and novel; the scoring window begins at 500k tokens, with progress before that shown for context only
- Harness notes
- The curve averages runs within each configuration and weights the 11 tasks equally; 15 configurations are published, cost is mean API cost per run, and only models with a valid run in every configuration are included
- Authors
- Håvard Tveit Ihle, Norwegian Defence Research Establishment (NDRE); API costs funded primarily by Epoch AI, secondarily by METR
- Released
- 2026-09-18
- License
- CC BY 4.0