Document parsers can score well overall and still miss tables, break formulas or scramble reading order, according to a benchmark of 12 systems across 1,250 held-out pages. The author’s leading open-weights approach combines a two-stage layout parser with a model specialising in prose, a useful reminder that document AI is not one problem with one score.
Watch Desk analysis
What happened
In a blog post published on 3 October, Ping Lin describes testing 12 document parsers and vision-language models on 1,250 held-out pages. The post says aggregate scores can conceal specific failures, including missing tables, broken formulas and scrambled reading order.
Lin’s reported best open-weights system combines a two-stage layout parser with a prose-specialist model. The post also says frontier hosted models perform better at text recognition but struggle with table detection. It gives no model names, individual scores or detailed test results in the supplied account, so the comparison is a useful signal rather than a buying chart.
Why it matters
A document system can extract plenty of words and still mishandle the information that makes a document useful: a table’s structure, a formula’s meaning or the order in which a page should be read. That matters wherever people rely on parsed documents for research, records or automated workflows. One headline score may not tell them whether the system can handle the pages they actually have.
The result also points to a practical design choice: combining a layout-focused parser with a prose specialist may work better on this test than asking one model to do everything. Whether that holds across other document collections remains an open question.
Our read
The important takeaway is not that one particular stack has won. It is that document-AI evaluations should show what goes wrong, not just how much text comes out the other end. If you are choosing a parser, test tables, formulas and reading order on your own documents before trusting an aggregate score. The tidy total is not always the tidy page.
What to watch
- Whether Lin publishes model-by-model scores and more detail on the evaluation setup.
- Whether the combined open-weights approach performs well on other document collections.
- How hosted models handle tables and layout in future comparisons.
Discussion spark: When choosing document AI, should teams prioritise a strong overall score or reliable handling of tables, formulas and reading order?
Sources and evidence
- Beyond OCR Scores: Where Document Parsers Fail (3 October 2026, 23:33 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.