Discussion

GPT-6.2 Sol nearly matches Opus 5.5 on score, at a fraction of the benchmark cost

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#14586

GPT-6.2 Sol scored 57.3 against Claude Opus 5.5’s 59.1 on Artificial Analysis’s Intelligence Index, while costing about a tenth as much per benchmark task. The comparison suggests the practical choice may hinge less on leaderboard rank than on whether your work rewards speed and lower bills, or Opus’s stronger results on some knowledge-heavy tests.

Watch Desk analysis

What happened

In a comparison published by OrcaRouter, both models were measured at their highest effort setting. The figures attributed to Artificial Analysis put the cost per Index task at $1.01 for Sol and $10.04 for Opus 5.5. Average task time was 8.0 minutes for Sol and 24.6 minutes for Opus.

The overall scores conceal differences between tests. Sol led on Terminal-Bench-Science and AutomationBench, while Opus scored higher on HLE-Diamond and the AA-Omniscience measure. On Terminal-Bench 4.0, the reported scores were close: 60.1% for Sol and 59.6% for Opus. These are results from the comparison’s benchmark setup, not a promise about every team’s workload.

Read OrcaRouter’s comparison.

Why it matters

A small lead on a composite score can look decisive until the time and cost of producing it turn up. Here, the reported task-cost gap is much larger than the score gap. For teams running many model tasks, that makes workload fit and the cost of checking answers part of the choice, not footnotes for the finance department.

The comparison also shows why “best model” is a slippery label. Sol’s reported advantage on scientific terminal work does not establish that it is better at every kind of analysis, just as Opus’s higher scores on selected tests do not settle the question for coding or routine agent work.

Our read

This is a useful case for choosing models by job rather than by crown. If you are deciding between them, compare both on tasks you can check and record the cost and elapsed time of completed work. A benchmark is a map, not the territory, but this one at least puts the tolls on the route.

What to watch

  • Whether later measurements reproduce the reported score, time and cost differences.
  • How the models compare at effort settings teams would actually deploy.
  • Whether results on specific tasks predict performance in real workflows.

Discussion spark: Would you choose the much cheaper, faster model when it nearly matches the overall score, or pay more for the strengths of a model that leads on selected tests?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.