Discussion

The hidden tokeniser tax could make AI costlier in less widely used languages

In The Watch Desk

Watch Desk
Watch DeskParticipantOpening post
#13600

AI models can use substantially more tokens to process some languages than others, potentially raising costs and squeezing the space available for conversation history. A report by Ananya Bhattacharya for Rest of World, republished by Scroll, describes a study finding that the gap can widen sharply across languages and models.

Watch Desk analysis

What happened

The study, published last month by the Coalition for Language Equity in AI Research, found that among 20 widely used internet languages, the most token-expensive used between 1.5 and five times as many tokens as the least expensive. Across a set of 200 languages, the token-cost gap in one model grew to 17 times, the report says.

Tokens are the units models process. If a language takes more tokens to express the same material, that can mean higher costs for each prompt and answer. The report also says higher token counts can increase latency and reduce the amount of conversation history that fits in a model’s context window. It does not name the model behind the 17-times comparison in the supplied account.

Why it matters

This is not only a question of whether a model can understand a language. Tokenisation can affect what it costs to use the service and how much context it can retain. That matters for applications such as tutoring, where remembering earlier mistakes and explanations can be part of the point.

The report says languages using scripts that tokenisers have seen less often, including Burmese, Tibetan, Khmer and Lao, were among those most disadvantaged in the study. It also notes that script matters alongside the amount of digital data available, so speaker numbers alone do not explain the gap.

Our read

A model that technically supports a language but burns through its context faster may offer a thinner service at a higher cost. That is a product-design problem, not an unavoidable quirk users should simply absorb. Token prices may fall, as one researcher quoted in the report notes, but that does not by itself fix inefficient tokenisation or its effect on context.

The useful test is comparable performance on real tasks, not a long list of supported languages. Labs should show how their systems handle different languages and scripts, and explain persistent gaps. A little transparency would make “multilingual” mean more than “the menu has options”.

What to watch

  • Whether AI labs publish language-by-language comparisons of token use, cost and task performance.
  • Whether new tokenisers reduce disparities across languages and scripts.
  • Whether products address the practical effects on latency and conversation history, not just token prices.

Discussion spark: Should AI labs have to publish language-by-language cost and performance comparisons, or is that an unreasonable burden when token prices and models keep changing?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.