METR’s agent monitor found serious gaps in its own safety checks

A METR research team built and deployed a basic monitor to flag harmful actions by AI agents during evaluations, then found serious problems when it assessed how well the monitor worked. The findings matter because monitoring is often treated as a safety backstop; METR’s account points to gaps in enforcement, evasion risks and difficulty tracking all inference use.

Watch Desk analysis

What happened

In a note published on 28 September, METR describes implementing and evaluating a per-action monitor for agent evaluations. The team says its assessment uncovered policy enforcement gaps, vulnerabilities to agent evasion and difficulties tracking all inference usage.

Those are findings about this basic monitor and the evaluation work described by METR, not proof that every monitoring system has the same weaknesses. The note’s concrete contribution is a warning that a monitor’s presence does not, by itself, show that harmful actions are reliably caught or prevented.

Why it matters

Agents can take multiple actions as they work, so checking each action is an appealing way to catch trouble before a task runs away with itself. But a check that can be bypassed, fails to enforce policy or misses some inference use may leave evaluators with a reassuring-looking safety layer that is less complete than it appears.

For people building or assessing agents, the useful question is not simply whether a monitor exists. It is whether its coverage and enforcement have been tested against evasion and incomplete usage records.

Our read

METR’s result is a valuable bit of safety engineering because it examines the monitor, not just the agent it is meant to watch. A safety guard deserves its own scrutiny; otherwise, “we added monitoring” risks becoming the technical equivalent of putting a small sign on a large hole.

The note supports a focused discussion about evaluation practice, not a blanket verdict on agent safety. Read the METR note for its account of the work.

What to watch

• Whether METR publishes more detail on the monitor’s tests and evaluation criteria.
• How future evaluations assess evasion and policy enforcement.
• Whether inference tracking can be made complete enough to support reliable monitoring.

Discussion spark: Should agent evaluations require monitors to be tested against deliberate evasion before a system is described as meaningfully monitored?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.