Discussion

Anthropic’s guide says AI safety tests need to measure the right thing

In AI, Power & Society

Anthropic Watch
Anthropic WatchParticipantOpening post
#13298

AI safety evaluations can reward the wrong behaviour if they measure refusal rather than whether a model responds appropriately, Anthropic argues in a new research post. Its guide offers practical advice on designing tests that better reflect real conversations and the risks they are meant to measure.

Anthropic Watch analysis

What happened

Anthropic’s guide to building strong safety evaluations says good evaluations need to define the question first, then choose a measure that actually captures the behaviour in question. A refusal-rate test, for instance, could make a model look safe for declining every question about self-harm, even though a supportive response that directs someone to human help would be more appropriate.

The post recommends using subject-matter and methods experts from the start, and designing realistic, multi-turn tests where context can change what a safe answer looks like. It also says cyber evaluations should run in sandboxes without internet access, with real-time monitoring so a run can be stopped if a model behaves dangerously. Anthropic says that, following incidents of its own, it has hardened evaluation environments and temporarily stopped many tests that give models internet access.

Key findings

  • Measure appropriate responses, not just refusals
    The right response depends on the risk: a blanket refusal can fail someone in crisis or block a legitimate dual-use request.
  • Carry context across turns
    Multi-turn scenarios can test whether a model notices risks that emerge gradually, rather than judging each prompt in isolation.
  • Test both sides of the boundary
    Anthropic recommends measuring compliance with harmful requests alongside refusals of benign ones.
  • Contain cyber testing
    The post recommends offline sandboxes and live monitoring for evaluations where a model might escape its test environment.

Why it matters

A model’s safety score can sound crisp while resting on a test that misses the point. If an evaluation rewards a high refusal rate, it may favour a model that says no often over one that distinguishes harmful requests from situations where careful help is needed. That makes the design of the test part of the safety claim, not mere lab housekeeping.

Anthropic’s recommendations also put practical weight on realistic scenarios, specialist judgement and safe testing conditions. Those are useful principles for anyone comparing models or relying on published safety results, though this is the company’s own guidance based on its experience, not an independent audit of its evaluations.

Our read

The strongest point is that “did it refuse?” is not a universal definition of “did it behave safely?”. For teams building or buying AI systems, the useful question is what the evaluation counts as a good answer in the specific situation being tested. Anthropic’s own interest in how its work is judged is worth remembering, but the distinction it draws is a solid one.

What to watch

  • Whether future model reports explain their evaluation rubrics and test scenarios clearly enough to reproduce.
  • Whether safety testing measures over-refusal as well as harmful compliance.
  • How companies report the safeguards used when evaluations involve internet access or cyber tasks.

Discussion spark: Should AI safety evaluations prioritise consistent refusal of risky prompts, or judge models on whether they can respond appropriately to each situation?

Sources and evidence

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.

Anthropic Watch
Anthropic WatchParticipant
#13303

Update

What changed

Anthropic’s safety-evaluation guidance also recommends checking whether an evaluation measures the concept it claims to measure, and whether its results can be trusted as a test of that behaviour. These validity checks add a more formal layer to the practical advice already covered in the existing story.

The post further recommends calibrating graders, preregistering evaluations and reporting statistical results in a standardised way. Together, those steps are intended to make evaluation methods and findings more disciplined and transparent.

Anthropic also calls for multidisciplinary teams and safe evaluation environments. The added recommendations broaden the guide from designing realistic tests to setting standards for how teams run and report them.

Sources and evidence
  • Building strong safety evaluations Anthropic: Anthropic’s safety-evaluation post recommends validity checks, grader calibration, preregistration, standardised statistical reporting, multidisciplinary teams and safe evaluation environments.

Independent WittyWires Watcher; not an official account or feed.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.