Elorian has launched Glimpse, a visual-reasoning model, alongside DecoyCount, a benchmark designed to test whether AI models count what is actually visible rather than rely on visual shortcuts. Glimpse scored 50.48 per cent in a company-reported test, the only model in the comparison to pass 50 per cent, though the runs were not directly matched.
Watch Desk analysis
What happened
DecoyCount tests how models count objects when images contain misleading cues, such as a reflection that could be mistaken for a second object. The benchmark was released with AI data company Mercor. In the accompanying paper, the highest reported score among frontier models was 43.17 per cent for GPT-6 Astra. Gemini-3.8-flash scored 32.71 per cent, Qwen3.8-max 28.31 per cent and Claude-Fable-5.1 25.22 per cent.
Elorian chief executive Andrew Dai told Latent Space that Glimpse scored 50.48 per cent on the 1,660-item test. He said the evaluation images and text were withheld from its training data and that the company’s contamination check found no overlap in the data it checked. Glimpse’s result came from one run; the other models’ reported scores were averages across three runs.
Why it matters
A model can miscount because it fails to see an object, or because it sees an object and reasons badly about what it represents. DecoyCount is intended to make those failures easier to distinguish, which could help developers identify what needs fixing rather than treating every wrong answer as the same problem.
That matters beyond puzzles. A warehouse system counting items, for instance, needs to distinguish what is in a box from what a label or familiar arrangement suggests ought to be there. Visual reasoning is still a long way from being settled by one benchmark, but this test points at a practical weakness worth measuring.
Our read
The result is intriguing, not a clean leaderboard victory: Glimpse’s single run sits beside three-run averages, and its score is reported by its own chief executive. Still, the benchmark asks a useful question about whether models can reason from visual evidence instead of guessing from patterns. That is a better test of progress than another model claiming to understand the physical world because it has seen a great many pictures.
What to watch
- Whether independent researchers reproduce the DecoyCount results across repeated runs.
- Whether the benchmark’s diagnostic categories help explain specific visual failures.
- Whether Glimpse’s approach improves results on practical tasks beyond counting.
Discussion spark: Should a new model be compared on a single run if the established models are averaged across three, or should benchmarks require matched testing before publishing a leaderboard?
Sources and evidence
- Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning (29 October 2026, 16:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.