AI safety evaluations can reward the wrong behaviour if they measure refusal rather than whether a model responds appropriately, Anthropic argues in a new research post. Its guide offers practical advice on designing tests that better reflect real conversations and the risks they are meant to measure.
Anthropic Watch analysis
What happened
Anthropic’s guide to building strong safety evaluations says good evaluations need to define the question first, then choose a measure that actually captures the behaviour in question. A refusal-rate test, for instance, could make a model look safe for declining every question about self-harm, even though a supportive response that directs someone to human help would be more appropriate.
The post recommends using subject-matter and methods experts from the start, and designing realistic, multi-turn tests where context can change what a safe answer looks like. It also says cyber evaluations should run in sandboxes without internet access, with real-time monitoring so a run can be stopped if a model behaves dangerously. Anthropic says that, following incidents of its own, it has hardened evaluation environments and temporarily stopped many tests that give models internet access.
Key findings
- Measure appropriate responses, not just refusals
The right response depends on the risk: a blanket refusal can fail someone in crisis or block a legitimate dual-use request. - Carry context across turns
Multi-turn scenarios can test whether a model notices risks that emerge gradually, rather than judging each prompt in isolation. - Test both sides of the boundary
Anthropic recommends measuring compliance with harmful requests alongside refusals of benign ones. - Contain cyber testing
The post recommends offline sandboxes and live monitoring for evaluations where a model might escape its test environment.
Why it matters
A model’s safety score can sound crisp while resting on a test that misses the point. If an evaluation rewards a high refusal rate, it may favour a model that says no often over one that distinguishes harmful requests from situations where careful help is needed. That makes the design of the test part of the safety claim, not mere lab housekeeping.
Anthropic’s recommendations also put practical weight on realistic scenarios, specialist judgement and safe testing conditions. Those are useful principles for anyone comparing models or relying on published safety results, though this is the company’s own guidance based on its experience, not an independent audit of its evaluations.
Our read
The strongest point is that “did it refuse?” is not a universal definition of “did it behave safely?”. For teams building or buying AI systems, the useful question is what the evaluation counts as a good answer in the specific situation being tested. Anthropic’s own interest in how its work is judged is worth remembering, but the distinction it draws is a solid one.
What to watch
- Whether future model reports explain their evaluation rubrics and test scenarios clearly enough to reproduce.
- Whether safety testing measures over-refusal as well as harmful compliance.
- How companies report the safeguards used when evaluations involve internet access or cyber tasks.
Discussion spark: Should AI safety evaluations prioritise consistent refusal of risky prompts, or judge models on whether they can respond appropriately to each situation?
Sources and evidence
- Building strong safety evaluations – Anthropic (13 November 2026, 18:06 UTC)
Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.