A ClickHouse benchmark found that AI agents often fail to select the right data, but giving them ranked table suggestions did not improve accuracy across 20 models. The findings also show how a model-based classifier can help audit failures, including answers a benchmark grader may have wrongly marked incorrect.
ClickHouse Watch analysis
What happened
In a post published on 27 October, ClickHouse describes testing Jev, TypeSafe’s System One model, on an agentic analytics benchmark. Jev does not generate text: it answers typed questions about supplied data, including which warehouse tables a question needs. ClickHouse says it ranked tables with an nDCG@10 score of 0.951 on a development set, taking about 0.4 seconds per question, without fine-tuning or gold data at run time.
But a good ranking was not a magic prompt. Feeding those rankings back to agents as hints left accuracy flat across 20 models, and turn savings from the development set did not hold up on the separate benchmark board. ClickHouse says the benchmark has 201 questions, while its experiments used a separate 136-question development set.
The post also describes using Jev to classify failures and check whether apparently failed answers might actually be correct. Across 277 failures in a four-model development run, the classifier flagged 37 as possible grader errors; a confidence threshold narrowed that to 18. ClickHouse traced high-confidence cases to a comparator that can reject answers based on row-count differences before checking whether columns are linked correctly.
Why it matters
Finding relevant data is a real weakness in text-to-SQL agents, but identifying the weakness is not the same as fixing it. ClickHouse’s results suggest table rankings can be useful for diagnosis while still failing to improve an agent when delivered as an in-loop hint. That distinction matters to anyone choosing retrieval tools based on a polished ranking score.
The grader audit points to another less glamorous source of benchmark error: the answer key and comparison logic can be wrong too. A model may look worse because it failed, or because the test failed to recognise a valid answer. Those are rather different problems to take back to the drawing board.
Our read
The most useful result is the uncomfortable one: a strong retrieval metric did not translate into better agent performance. ClickHouse’s post is a company-authored account of its own benchmark work, so treat the figures as reported findings, not a universal verdict on retrieval methods. Still, it makes a persuasive case for testing the whole loop, including the grader, rather than stopping at a tidy ranking score.
What to watch
- Whether ClickHouse publishes results from further tests of the in-loop hint.
- How the comparator bug changes scores when corrected.
- Whether the table-ranking results hold up on larger, less familiar warehouses.
Discussion spark: If a retrieval tool ranks the right tables but does not improve the agent’s accuracy, is it still useful enough to deploy, or should the team spend its effort elsewhere?
Sources and evidence
- Source update (27 October 2026, 16:57 UTC)
not affiliated with or endorsed by ClickHouse