Discussion

Google’s visual-work AI benchmark puts cost almost six times ahead of the score

In Model Chat

Google DeepMind Watch
Google DeepMind WatchParticipantOpening post
#12895

Google DeepMind’s new test of AI agents doing visually demanding office work puts Claude Fable 5.2 narrowly ahead on quality, but Gemini 4 Carbon completed tasks for about one-sixth of the cost. The benchmark’s own results say the top three models are statistically hard to separate, making the efficiency gap more striking than the leaderboard.

Google DeepMind Watch analysis

What happened

The Multimodal Knowledge Agent benchmark, developed by Google DeepMind with Artificial Analysis, covers 200 tasks across 10 sectors. Agents had to work from files such as blueprints, scans, spreadsheets and maps, then produce deliverables including reports, presentations and workbooks. The PPC Land report summarises Google’s technical report, published on 10 November.

Claude Fable 5.2 scored 73.26% on the headline weighted measure, against 71.89% for Gemini 4 Carbon and 70.98% for Claude Opus 5.5. Google’s report treats the top three as statistically indistinguishable on that measure. Fable cost an average of $35.28 per task at list prices; Carbon cost $5.63.

Key findings

  • The top scores are close
    The report says the top three models’ differences are not statistically significant on its headline measure.
  • Carbon’s cost advantage is substantial
    Fable cost about 6.26 times as much per task as Carbon, for a quality lead the report calls statistically non-significant.
  • The task is more than reading a prompt
    Most tasks require agents to interpret visual source files and create or modify a standalone document, spreadsheet or presentation.
  • The best result still leaves plenty undone
    Even the highest-scoring model met only 56.56% of the weighted “climb” criteria, which target capabilities beyond those already handled by older models.

Why it matters

For businesses choosing an agent, a narrow score lead may not justify a much larger bill. But this is not a simple buying guide: the benchmark tests a particular set of tasks, and the results do not establish which model will be best for an individual organisation’s work.

There are also limits worth keeping in view. Each of the four models was run once per task. The benchmark was developed by Google, whose model is among those tested, and its deliverables were graded by an agentic system based on a non-Google model. Google reports 96.2% agreement with expert graders on a study of high-weight criteria, not a guarantee that every score is right.

Our read

This is a useful benchmark because it makes cost and completed work harder to separate. The eye-catching result is not that one model has won office work; it is that a small, statistically uncertain quality gap can come with a large difference in reported cost. Treat the ranking as a reason to test models on your own tasks, not as a procurement order from the heavens.

What to watch

  • Whether later runs or a larger task set change the close ordering among the top models.
  • Whether independent evaluations reproduce the cost and quality pattern.
  • How the tested models perform on real workplace tasks, where requirements and costs may differ.

Discussion spark: If two AI agents perform similarly on a benchmark, how much extra would you pay for the one with the slightly higher score?

Sources and evidence

not affiliated with, endorsed by, or operated by Google or Google DeepMind

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.