METR says AI is speeding up Anthropic’s research, but redactions leave key evidence under wraps

METR’s assessment says AI has helped Anthropic’s models improve roughly twice as quickly as a year earlier, with much of the acceleration attributed to AI assistance. The findings also come with a catch: Anthropic redacted much of the evidence in the public report, leaving readers reliant on METR’s judgement.

Anthropic Watch analysis

What happened

The Verge reports that METR spent 11 weeks assessing Anthropic’s AI research and development. METR says benchmark scores point to roughly two years of historical progress in the past year, and that AI’s contribution to the acceleration was growing. The researchers said they found no clear sign of diminishing returns or bottlenecks that would prevent faster progress.

METR also reports that Anthropic staff using its most capable internal model said they were, on average, around 12 times as productive as without AI assistance. That is employees’ self-reported productivity, not a controlled measure of output. The report describes three unreleased models completing training between July and October.

The Verge says Anthropic was allowed to redact parts of METR’s report and did so extensively. METR says the public version omits a majority of the evidence it considered most informative. Anthropic co-founder Sam McCandlish defended the redactions, saying the details could help competitors accelerate their work and increase the risks METR is concerned about.

Why it matters

If AI is helping researchers build better AI, the pace of progress becomes a question of more than larger models and bigger computing budgets. METR’s findings point to AI assistance as a growing factor, while the redactions make it harder for outsiders to judge how strong the evidence is or what uncertainties remain.

Our read

This is a consequential warning about both acceleration and accountability, not proof that AI is autonomously improving itself. The reported productivity figures are self-reported, and the public cannot inspect much of the evidence METR says informed its conclusions. The argument over whether those details should be public is now part of the story, rather than a footnote.

What to watch

• Anthropic’s promised report on the pace of AI progress and planned safety steps. – Whether METR or other independent researchers can publish more of the evidence behind the assessment. – Whether future evaluations measure AI’s contribution to research in ways outsiders can scrutinise.

Discussion spark: When a company restricts evidence in an independent assessment to avoid helping competitors, is that a reasonable safety trade-off or does it make the findings too hard to scrutinise?

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.