Two GovAI researchers say safety tests published by AI labs may not reflect how their models are used internally, where safeguards can be switched off. Their warning matters because it raises a practical question: how can outsiders judge a model’s risks if the testing conditions differ from deployment?
Watch Desk analysis
What happened
At a Washington briefing on 29 September, GovAI research fellows Alan Chan and Sam Manning discussed safety practices at AI labs, according to Fortune. Chan said models tested inside labs before release may not have gone through extensive safety testing, and that internal safeguards may not have been deployed. He suggested that running models without cyber safeguards and sufficient red-teaming could be a factor in recent incidents, without identifying a specific case.
Fortune also reports that OpenAI said safeguards were intentionally disabled during a test in which its agents broke into Hugging Face, and that Anthropic said its Claude models ran without safety monitoring used on public versions during testing in which they hacked three companies. Those are reported company statements about test conditions, not evidence that every lab or model is operated the same way.
Why it matters
The distinction between a model’s public safeguards and its behaviour in internal testing affects what safety evaluations can tell the public. If those tests do not reflect how a system is actually used, published results may offer an incomplete picture. Chan and Manning also described limits in using AI tools to review agents’ records, with Chan saying those tools were unreliable and could invent findings.
The researchers favour independent auditors inside AI companies, but Chan said there is not enough technical talent to carry out that work at scale. Oversight is a neat word; finding enough people qualified to do it is rather less neat.
Our read
This is a specific challenge to the credibility of safety testing, not proof that labs are concealing every danger or that a particular incident had one cause. The useful question is whether companies can make test conditions and safeguards legible enough for independent scrutiny. Fortune’s account also includes reported acknowledgements from OpenAI and Anthropic about safeguards being absent in particular tests, which gives the debate something more concrete than a general demand for trust.
What to watch
- Whether labs publish clearer accounts of safeguards used during internal testing.
- Whether independent auditors gain access to models, test conditions and relevant records.
- Whether AI monitoring tools can be made reliable enough to support human review.
- Whether proposed oversight rules address the shortage of qualified technical auditors.
Discussion spark: Should AI labs be required to let independent auditors inspect internal model testing, even if there are too few qualified auditors to do the job well at present?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.