Public AI benchmark scores can be distorted by flaws in the tests meant to measure them. Prime Intellect says its audit found brittle grading and opportunities for reward hacking in several popular evaluations, and its fixes offer a practical route to making at least some of those tests harder to game.
Watch Desk analysis
What happened
In a post titled “Learnings from Auditing Public Evaluations”, Prime Intellect describes examining AutomationBench, OfficeQA Pro v2 and HLE-Diamond. It says the team found design patterns and grading logic that could skew reported model capabilities.
The post gives two concrete examples: AutomationBench asks models to describe requests in prose rather than using JSON schemas, while OfficeQA Pro v2 relies on brittle regular-expression extraction that can reject valid answers. Prime Intellect says it open-sourced fixes, including schema-based requests and synchronous monitors intended to detect web-search leaks.
Why it matters
A benchmark is only as persuasive as its rules and grader. If a test rewards a workaround, or marks a valid response wrong because its extraction code is too fussy, the resulting score can say more about the test design than a model’s capabilities. That matters when public evaluations are used to compare systems or support claims about progress.
Prime Intellect says the fixes pushed the affected benchmarks close to saturation. That is a useful warning as well as a repair: once a test becomes easy to max out, it may no longer distinguish the systems being compared. The post argues that private evaluations are more reliable than depending on third-party public metrics, but that is the team’s conclusion, not a universal result established by these examples alone.
Our read
The best part of this audit is its specificity. “Benchmarks can be gamed” is familiar; showing how a prose-only interface or a brittle answer parser can skew results gives evaluators something concrete to inspect and improve. The fixes are a welcome contribution, though open code is the start of scrutiny, not a magic stamp of validity. Readers comparing model scores should ask what the benchmark actually tests, how answers are graded and whether the evaluation still has room to separate strong systems.
What to watch
- Whether the benchmark maintainers adopt and document the proposed fixes.
- Whether the changes reduce reward hacking without creating new ways to game the tests.
- Whether other evaluations publish similarly specific audits and grader code.
Discussion spark: Should public AI benchmarks remain the main scoreboard if their rules and graders can be inspected and patched, or do credible comparisons need private evaluations too?
Sources and evidence
- Learnings from Auditing Public Evaluations (3 November 2026, 18:37 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.