Anthropic gives METR deep access to test its AI models, raising questions about independence

Anthropic has chosen the nonprofit METR to assess its AI models, giving six of the group’s staff “employee-like access” to its models and tools. The arrangement offers outside evaluators unusually deep access, while leaving questions about how independent that oversight can be.

Anthropic Watch analysis

What happened

The San Francisco Standard reports that METR staff will be able to test Anthropic’s models and write reports intended for public release. Anthropic says it cannot veto their conclusions, but can redact material it considers sensitive, such as customer data or details about model structure. METR will also have a place in Anthropic’s San Francisco office.

The report describes ties between the labs and the safety community: METR’s founder previously worked at DeepMind and OpenAI, and its staff includes former employees of leading AI labs, including Anthropic. Anthropic says deep access through trusted evaluators helps it share evidence about safety while protecting sensitive information.

Why it matters

External testing can reveal risks a company’s own teams miss. But access is not the same as independence: the terms of the relationship, the evaluator’s connections and the company’s power to redact all affect what the public can learn.

The Standard also reports that critics see close ties across the field as a potential threat to independence. Others quoted in the article argue that previous employment does not automatically compromise an evaluator. The practical question is whether the arrangement produces useful scrutiny, and whether readers can see enough of the evidence to judge it.

Our read

This is more substantial than a safety pledge: METR gets access to models and tools, and Anthropic says the group can publish conclusions it cannot veto. The test is what those reports disclose, what gets redacted and whether outside scrutiny can challenge the company’s own account. “Independent” is a description that has to earn its keep.

What to watch

• What METR’s first reports say, and how much detail they contain.
• How often Anthropic uses its right to redact material.
• Whether other evaluators get comparable access, and under what terms.

Discussion spark: Should AI labs choose and fund the outside groups that assess their models, or should independent oversight require evaluators selected and supported by someone else?

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.