Discussion

Dharma AI says its 2B document model beat frontier models on its extraction benchmark

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#11297

Dharma AI says its 2-billion-parameter Field Extractor scored higher than four tested frontier models on a document-extraction benchmark, while costing less to run in the company’s comparison. If the result holds beyond that test, it is a useful case for choosing a model built for a specific job rather than reaching automatically for the biggest general-purpose system.

Watch Desk analysis

What happened

The Field Extractor turns document images into structured JSON. It can either find fields without a supplied schema or fill a schema supplied by the user, including returning null for requested fields that are absent from the document. Dharma AI says the same checkpoint handles both modes without per-document-type fine-tuning.

In the company’s benchmark of 546 documents and 1,638 test cases, the model scored 0.834 on its field-level measure. Dharma AI says the closest tested frontier configuration scored 0.816 and the lowest 0.741. It reports estimated running costs between about six and 147 times lower than those APIs. The company’s comparison uses a self-hosted L40S GPU at $2.24 an hour; it is an estimate of compute cost, not a sale price.

Why it matters

Invoices, forms and identity documents often need specific values passed into another system, not a faithful transcription of every line. A model that returns typed fields in the required shape, and leaves missing values blank instead of inventing them, could cut both processing work and the cost of high-volume extraction.

The result is a company-reported benchmark, not a guarantee that the model will win on every document set. Its score combines field matching, value similarity and valid JSON output, so the exact result depends on the test and its scoring rules.

Our read

This is a credible argument for trying task-specialised models before paying frontier API rates for routine extraction. The useful next step is a pilot on your own documents, with checks for accuracy, exceptions and the real cost of running the system. Benchmarks are a starting line, not a procurement decision wearing a medal.

What to watch

  • Whether independent tests reproduce the quality and cost gap.
  • How the model performs on documents and languages outside the reported benchmark.
  • Whether users can deploy it with the hardware, latency and safeguards their workloads require.

Discussion spark: For document extraction, would you choose a smaller task-specialised model on the strength of a benchmark like this, or require it to prove itself on your own documents first?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.