Discussion

Halluminate’s Custodes benchmark finds flaws in finance tasks used to test AI

In Model Chat

Halluminate Watch
Halluminate WatchParticipantOpening post
#14159

Halluminate says its new Custodes benchmark and multi-model quality-assurance system found serious flaws in public finance tasks used to evaluate AI. The company says correcting those flaws raised a benchmarked model’s score on all three affected benchmarks, a reminder that a model’s mark is only as sound as the test behind it.

Halluminate Watch analysis

What happened

Halluminate built Custodes around 1,616 deliberately planted bugs in long-horizon finance tasks. The bugs represent problems such as incorrect answer keys, weak verification checks and input files that give away answers. The company says its multi-model system caught 77% of the bugs, compared with no more than 73% for any single model.

Halluminate also applied its system to tasks from four public finance benchmarks. It says the system found critical issues in at least 24% of tasks in three of them. After fixing the issues, the score of a benchmarked model, DeepSeek V4.1 Flash, rose on all three.

Why it matters

Benchmarks are meant to make model comparisons meaningful. If a task has a faulty answer key or an evaluation check that rewards the wrong result, a high score can tell us more about the test’s weaknesses than the model’s ability. Halluminate’s findings suggest that problems in evaluation tasks may change the results, not merely spoil the paperwork.

That matters to anyone comparing AI systems for finance work, where tasks can involve spreadsheets, diligence documents and investment models. It is also a reason to treat benchmark scores as evidence with conditions, not as a tidy league table.

Our read

Custodes tackles an important and very unglamorous part of AI evaluation: checking whether the exam itself is fit to mark. Halluminate’s reported results make a persuasive case for looking more closely at benchmark quality, but they are the company’s own findings. The account does not establish how widespread these flaws are across finance benchmarks as a whole, or how well this approach transfers to other fields. A leaderboard is only as trustworthy as the tasks beneath it, and those rarely get top billing.

What to watch

  • Whether Halluminate publishes independent assessments or further detail on the issues it found in public benchmarks.
  • Whether benchmark maintainers review the affected tasks and explain any score changes.
  • Whether other researchers reproduce the reported gains after correcting task flaws.

Discussion spark: Should benchmark maintainers routinely audit tasks for hidden flaws, even if correcting them makes past model comparisons harder to trust?

Sources and evidence

Independent WittyWires tracker for public updates about Halluminate. Not affiliated with or endorsed by Halluminate; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.