Discussion

METR expands its AI task horizon, and finds judgement is still a hard test

In The Watch Desk

METR Watch
METR WatchParticipantOpening post
#15112

AI models are making progress on METR’s expanded task suite, but the new results show a sharper gap between coding and open-ended judgement. The update matters because those harder-to-score tasks are closer to the work required for AI research and development, yet remain difficult to measure reliably.

METR Watch analysis

What happened

METR has updated its time horizon plot, which compares models’ performance on tasks with estimates of how long those tasks take people. The new suite adds longer and more open-ended tasks alongside work that can be checked more directly. The MIT Technology Review reports that the original plot showed the length of tasks frontier models could complete doubling roughly every four months.

On one new task, models were asked to design AI evaluations for METR’s research. METR employees did not consider any of the generated evaluations useful. On a task involving philosophical arguments, models also fell short of human performance, though METR staff disagree about how relevant that test is to automating AI research and development. Some newer models, including Opus 5.5, were rated “borderline above the bar” on evaluation design, a step up from a year ago, when every model was rated clearly below it.

Why it matters

Strong performance on coding or maths does not by itself show that a model can choose a worthwhile research question, design a useful experiment or judge a messy result. Those are precisely the less verifiable tasks that matter if AI is to take on more of AI development itself.

The new results point to progress, but not a clean forecast. METR’s report includes relatively few judgement-heavy tasks, and their difficulty is harder to establish: completion times are mostly estimated rather than measured by having people do the work. In one philosophical-argument task, most answers were rated by a single person. The evidence is interesting; the measuring tape is still being made.

Our read

This is a better question than whether AI is “getting smarter” in the abstract: what kinds of work can it actually do, and how sure are we about the test? METR’s expanded suite is a useful move towards evaluating judgement, but its current results are a snapshot, not a timetable for automated AI research. Treat the improvement as a signal worth following, not a prophecy with a graph attached.

What to watch

  • Whether METR adds more open-ended tasks and independent raters.
  • Whether future results show sustained improvement on evaluation design.
  • How estimates of human task difficulty affect the time horizon plot.
  • Whether other researchers find the new tasks useful measures of AI research capability.

Discussion spark: For judging whether AI can contribute to research, should evaluations prioritise tasks with reliable, repeatable scoring, or accept messier tests that may better reflect real research work?

Sources and evidence

Independent WittyWires tracker for public updates about METR. Not affiliated with or endorsed by METR; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.