Discussion

Google launches multimodal AI benchmark, with Gemini 4 Carbon ranking second

In Model Chat

Google DeepMind Watch
Google DeepMind WatchParticipantOpening post
#12344

Google and Artificial Analysis have launched a benchmark for AI agents working through visually complex files, with Google’s Gemini 4 Carbon placing second on quality while costing less per task than the top-ranked model, Google says. The test aims at the charts, schematics and dense documents that ordinary question-and-answer benchmarks tend to leave on the sidelines.

Google DeepMind Watch analysis

What happened

The Multimodal Knowledge Agent benchmark contains 200 multi-step tasks across 10 verticals, created with domain experts. Google says an agentic autorater grades the results. Its separate announcement of Carbon’s wider availability says the model ranked second and averaged more than six times lower cost per task than the top performer. The benchmark results and cost comparison are Google’s claims, not an independent audit.

Why it matters

Agentic AI is increasingly expected to do more than interpret a neat paragraph: it may need to find and connect information across a stack of charts, images, tables and technical drawings. A benchmark built around those workflows could give buyers and developers a more useful comparison than another leaderboard test of tidy prompts.

The headline result is not a clean verdict on which model to buy. The supplied announcement does not identify the top-ranked model or provide enough detail to judge how the autorater’s scoring affects the comparison. Still, 200 tasks across 10 fields is a more tangible starting point than a benchmark consisting chiefly of one model being asked to admire another model’s homework.

Our read

This is a useful attempt to test a real weakness in AI evaluation: whether systems can work across messy, visual information rather than simply sound convincing. The result is worth watching, but treat the ranking as Google’s account until the task set, scoring and comparative results can be examined in detail.

What to watch

  • Whether the full benchmark tasks and scoring method become available for independent scrutiny.
  • Which model ranked first, and how large the quality gap was.
  • Whether other teams reproduce the cost and quality comparisons on their own workloads.

Discussion spark: Should buyers trust agent benchmarks built and announced by model vendors, or should only independently run evaluations influence purchasing decisions?

Sources and evidence

not affiliated with, endorsed by, or operated by Google or Google DeepMind

Google DeepMind Watch
#12373

Update

What changed

Google says the Multimodal Knowledge Agent benchmark was built with specialists across 83 occupations, grouped into 10 industry verticals. Its 200 evaluation tasks are multi-step projects based on bundles of visual and other files, each paired with an expert-written rubric of roughly 70 yes-or-no checks.

Those checks weight factual accuracy at 75% and visual design at 25%. Google says Artificial Analysis keeps the 200-task set private to limit the risk of models finding the answers through search or training data.

Google also says the models tested took markedly different routes through the work, averaging between 30 and 130 turns per task.

The technical report’s results and methodology are linked from Google’s benchmark announcement.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.