Claude Opus 5.5 leads Lilt’s multilingual coding benchmark, with the company reporting a small score gain over Opus 5 and lower task costs. The results suggest language coverage and the number of steps an agent takes can matter as much as the model’s headline price.
Watch Desk analysis
What happened
Lilt-org’s post on Hugging Face describes 324 agentic coding tasks across 10 languages, each designed with a native speaker. The team ran five trials per task using Harbor v0.21.0 and the Terminus-2 agent. It reports Opus 5.5 about three percentage points ahead of Opus 5, with the largest gains in Chinese and Korean and a slight regression in German.
Lilt-org also says Opus 5.5 used about 30% fewer tokens, making it roughly 45% cheaper per task. Its post says Gemini scored 2.5 points above GPT-6 Sol but cost seven times more per task. See the leaderboard.
Why it matters
The comparison puts a useful wrinkle in the usual model-price race: what matters to a user is the cost of completing a task, not just the price of each token. Testing in ten languages also gives a more varied view than an English-only coding contest, though these results cover this benchmark’s tasks and setup, not every developer’s workload.
Our read
This is a worthwhile benchmark to watch because it reports language-level differences and task costs, rather than handing out one shiny overall ranking and calling it a day. Lilt-org designed and ran the evaluation, so its results are a useful account of this test, not a universal verdict on coding models. Developers should compare models on their own languages, tools and jobs before treating the leaderboard as a procurement plan.
What to watch
- Whether Lilt-org publishes more detail on the task set and results by language.
- How Opus 5.5 performs in independent tests and real coding workflows.
- Whether lower token use translates into lower costs for other users and providers.
Discussion spark: When choosing a coding model for multilingual work, should teams prioritise performance across languages or the cost of completing their own tasks?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.