Firecrawl has published a benchmark for legal search APIs built around 581 requests for official documents. In its 3 November run, Firecrawl says its Government & Legal Index put the right official page first 76.2% of the time, ahead of Exa and Parallel, while keeping the service free to use.
Firecrawl Watch analysis
What happened
The benchmark covers searches for statutes, regulations, court opinions and agency guidance. It has 290 public requests, with answer keys and scoring code, and 291 held-out requests. Each question has a frozen answer based on an official source, and scoring matches URLs rather than asking a model to judge whether a result is correct.
In Firecrawl’s test, its index scored 76.2% for putting the official page first and 96.6% for finding it in the top ten. Exa scored 59.6% and 89.5% on those measures; Parallel scored 54.2% and 90.5%. A contradicting page appeared in the top ten for 9.6% of Firecrawl’s requests, statistically level with the competitors’ reported rates.
Firecrawl says the benchmark is built from recent customer searches, rewritten as equivalent questions, and refreshed weekly. The company says it has run the improvement loop eight times internally, and that this is the first published version. Read Firecrawl’s benchmark and methodology.
Why it matters
Legal research is a useful stress test for search: finding a page with similar wording is not the same as finding the official document. The benchmark makes that distinction measurable and publishes enough of its questions and scoring method for others to inspect.
The headline results are Firecrawl’s own, and its index leads on putting the official document first in this run. That is evidence about this benchmark, not a universal verdict on which search API is best for every task. The held-out questions and weekly refreshes matter because a fixed test can become a target once its answers are known.
Our read
This is more than a leaderboard with a confident hat: the public questions, frozen answer key and deterministic scoring make the comparison more useful to challenge. Firecrawl’s results are promising, especially on the top result, but the company is both benchmark-maker and competitor. The best next step is for other search teams to run the same test and for future versions to show whether improvements carry over.
What to watch
- Whether independent teams reproduce the rankings on the public set.
- Whether later benchmark versions show improvements on fresh, held-out searches.
- Whether the 16.6-point gap in top-result accuracy persists across future runs.
Discussion spark: Is a benchmark built from one company’s customer searches and scored against official documents a fair test of legal search, or does it give the benchmark-maker too much control over the contest?
Sources and evidence
- Source update (4 November 2026, 00:00 UTC)
not affiliated with or endorsed by Firecrawl