METR says Anthropic’s AI R&D warning may have been crossed earlier than its report suggests

METR’s review of Anthropic’s August risk report agrees that the direct catastrophic risk from models as of 15 July was low, but says Anthropic’s own early-warning threshold for AI-driven research acceleration may plausibly have been crossed. The disagreement is about how confidently the evidence supports that reassuring account, not a claim that a catastrophe is imminent.

METR Watch analysis

What happened

METR reviewed the “Risks from automated R&D” section of Anthropic’s public report. Anthropic said models had not met either of its thresholds for automated R&D: replacing its research staff at competitive cost, or dramatically accelerating AI progress, defined as at least a doubling attributable to automated research.

METR broadly accepts the case against full substitution. It is less persuaded by the argument that the acceleration threshold had not been crossed by 15 July. The review says Anthropic’s public report presents a leading-indicator estimate as below 2×, while figures in a private version imply about 1.97×, a calculation METR says it made from the report’s figures. METR’s own analysis produced a range of 1.3× to 2.8×, depending on the definition and period measured. The organisation stresses that there is substantial methodological uncertainty and uncertainty about how these indicators relate to downstream capabilities.

Why it matters

The threshold is an early warning, not proof of catastrophic risk. But a measure that can plausibly sit either side of its trigger deserves clearer explanation than a neat “less than two” suggests. The review also says Anthropic’s public account understated evidence for acceleration and uncertainty about what would have happened without AI assistance.

That makes this more than a technical argument over a decimal place. Companies setting their own safety thresholds need readers to understand when the evidence is close, contested or hard to measure.

Our read

METR’s review is a serious challenge to the confidence of Anthropic’s presentation, not a finding that its models had already automated AI research or posed high catastrophic risk. The useful takeaway is the gap between “the threshold was not crossed” and “it plausibly may have been”: that uncertainty belongs in the headline-level explanation, not in the small print.

What to watch

• Whether Anthropic clarifies how it calculated and communicated the acceleration estimate.
• How both organisations define the leading indicators and address their methodological uncertainty.
• Whether later assessments show sustained acceleration, and how much is attributed to AI assistance.

Discussion spark: When a safety threshold estimate is close to its trigger and the measurement range crosses it, should a company report the result as below threshold or as genuinely uncertain?

Independent WittyWires tracker for public updates about METR. Not affiliated with or endorsed by METR; this is not an official account.