MLCommons has published a proof of concept for testing a closed AI model without giving the model developer the benchmark prompts or giving the evaluator the model weights. The work brought MLCommons, Google DeepMind, OpenMined and AVERI together for a double-blind run of a reserved AILuminate safety benchmark.

MLCommons Watch analysis
What happened
The model ran inside a container through secure computation. MLCommons says the design protected the unseen test data and the proprietary model while still producing a reliability measurement. It was not simply a hidden question set. The evaluator and developer each kept hold of the thing the other side was not meant to inspect.
The reserved prompts were intended to be fresh to the model, reducing the chance that apparent performance came from prior exposure. Cryptographic controls then made access and disclosure harder by design rather than relying only on policy or promises.
Why it matters
Popular benchmarks can turn from measuring sticks into revision guides. Once a test set leaks, training and tuning can quietly favour its shape. A model may climb a leaderboard because it has rehearsed the exam, not because it will handle an unfamiliar real-world problem.
Double-blind evaluation gives benchmark operators, model developers and deployers a way to protect different secrets at the same time. A bank might need to test with sensitive data. A model lab may refuse to expose weights. The evaluator still needs confidence that the score came from the agreed test.
Our read
This is not a safety certificate, and one proof of concept cannot establish that every result is trustworthy. It is, however, a better answer than bolting another leaderboard to the wall and hoping nobody has smuggled in the answers. The controls make contamination, gaming and accidental disclosure structurally harder.
What to watch
- Whether the method can be repeated across other models and evaluation suites without making independent scrutiny impossible.
- How much of the process, threat model and failure handling becomes public enough for outsiders to challenge.
- Whether organisations adopt double-blind testing for consequential deployment decisions rather than treating it as a research demonstration.
Discussion spark: Would you trust an AI evaluation more if the test set, model weights and evaluator were technically isolated, or is public reproducibility still essential?
Sources and evidence
- The key to trustworthy AI evaluation is secrecy by design (27 August 2026, 13:13 UTC)
- MLCommons official publication feed (27 August 2026, 13:13 UTC)
Independent WittyWires tracker for public updates about MLCommons. Not affiliated with or endorsed by MLCommons; this is not an official account.