UK AI safety body backs open cards for reproducible benchmark results

The UK AI Security Institute is working with the EvalEval Coalition to publish AI evaluation results in a common, inspectable format. The important shift is practical: benchmark scores can now come with the setup details needed to reproduce and interpret them, rather than arriving as mysterious numbers in a slide deck.

Watch Desk analysis

What happened

EvalEval says AISI is using its infrastructure and Every Eval Ever schema to share evaluation methods, findings and configuration information through public Evaluation Cards. The collaboration covers five benchmarks in the paper’s main experiment: HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0.

The released material includes results for six frontier models, including Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4. Two related cyber evaluations, Cyber CTFs and The Last Ones, use a different and partly overlapping group of models.

EvalEval says the project is intended to address a familiar weakness in AI reporting. Results are often spread across different formats and platforms, with too little information to reproduce the run. Its Evaluation Cards bring benchmark metadata, evaluation-run data and model metadata into a shared record. That is less glamorous than announcing a new model, but considerably more useful when the leaderboard starts wearing a crown.

The collaboration also builds on AISI work including OptStop, which aims to make evaluations more efficient, and HiBayES, which applies hierarchical Bayesian modelling to evaluation. The source says the results accompany AISI’s paper, How Inference Compute Shapes Frontier LLM Evaluation.

Why it matters

A benchmark result is only as meaningful as the conditions that produced it. The supplied report says performance on Humanity’s Last Exam changes with evaluation protocol and inference-time compute. When models receive correctness feedback from an oracle after each attempt, they can continue solving additional tasks as token use increases.

That means two apparently similar scores may not be measuring the same thing. Open reporting of prompts, configurations, token budgets and evaluation procedures gives researchers a better chance of spotting those differences before a score becomes marketing folklore.

Our read

This is a quietly important piece of AI infrastructure. It does not make benchmarks perfect, and a shared card cannot rescue a poorly designed test. It does make the field’s working notes easier to inspect, compare and challenge.

Readers should treat Evaluation Cards as a useful evidence layer, not a universal truth machine. The best result is not necessarily the highest number. It is the one whose conditions are clear enough for somebody else to understand what happened and, ideally, try it again.

What to watch

• Adoption:
  whether other model developers and evaluation teams report results using Every Eval Ever.
• Method detail:
  whether the cards expose enough configuration information to reproduce each run.
• Protocol effects:
  how much scores move when inference budgets, feedback and prompting change.
• Coverage:
  whether future cards extend beyond the initial health, mathematics, coding and cyber evaluations. The specific WittyWires relevance is concrete: AISI and EvalEval are building open infrastructure for checking how frontier AI systems perform, and for showing the conditions behind their scores.

Discussion spark: Should AI benchmark results be treated as incomplete until their prompts, inference budgets and evaluation settings are published, or would that make comparisons too cumbersome to be useful?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.