Full Fact’s ten-week test finds chatbots still stumble on UK facts

Five leading chatbots were asked the same 22 UK-focused questions every day for ten weeks. Full Fact’s test found answers that could be correct one day, stale or inconsistent the next, with confident delivery regardless.

Watch Desk analysis

What happened

The fact-checking charity tested Grok, DeepSeek, Gemini, ChatGPT and Claude on civic questions about subjects including voting, asylum support and official statistics. Resultsense reports that Full Fact scored responses against five criteria and published the questions, benchmark answers and analysis as the start of a public benchmark it plans to continue.

One revealing error came when Gemini 2.5 Flash answered a question about small-boat arrivals in the first half of 2026. On 4 August, it gave 12,210 by subtracting July’s 2,316 arrivals from a running total of 14,526 that included 1 August. The arithmetic worked; the dates did not. Several models also gave inconsistent answers about England’s most recent major football trophy, while some cited the outdated £8.86 weekly asylum-support rate after it had risen to £9.95 in June 2025.

Why it matters

These are ordinary questions with practical consequences. A stale figure wrapped in a fluent answer can be harder to catch than an obvious mistake, particularly when someone is checking an entitlement, deadline or official statistic. Full Fact chief executive Chris Morris said the charity wants models to check live, authoritative sources for time-sensitive facts and explain where their citations come from.

The test also offers something more useful than a single dramatic chatbot failure: repeated checks across several models, using the same questions over time. Resultsense reports that OpenAI told The Independent accuracy is a priority and that it keeps working to reduce errors. The article says The Independent also sought comment from Anthropic, Google, DeepSeek and Grok’s developer.

Our read

The awkward lesson is not that chatbots are always wrong. It is that sounding certain is not a dependable signal that the answer is current, or even that its dates line up. A public, repeatable benchmark gives users and organisations a better basis for asking whether improvements are real. If you use a chatbot for anything time-sensitive, check its answer against the current authoritative source before acting on it. Confidence is a tone, not a citation.

What to watch

• Whether Full Fact continues the benchmark and publishes results over time.
• Whether model makers improve access to current authoritative sources and make citations easier to check.
• Whether the same failures persist on questions involving entitlements, deadlines and official figures.

Discussion spark: Should chatbots be expected to check live, authoritative sources before answering time-sensitive questions, or should users remain responsible for verifying every figure?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.