Cartesia proposes byte-level language models that learn where to chunk text

Cartesia researchers say a new language-model design can learn how to group text directly from bytes, rather than relying on a hand-built tokenizer. Their early results suggest gains in compute and data efficiency over standard tokenised Transformer baselines, with code among the difficult-to-tokenise areas they tested.

Watch Desk analysis

What happened

In a post published on 28 October, the Cartesia researchers describe hierarchical networks, or H-Nets. Instead of first converting text into fixed tokens, the models use dynamic, end-to-end chunking to decide how bytes should be grouped.

The researchers report efficiency gains in both one-stage and two-stage versions of the architecture. They also describe behaviours including in-context tokenisation and strong performance on difficult-to-tokenise domains such as code. Read Cartesia’s post.

Why it matters

Tokenisation is one of the quiet design choices shaping how language models process text. A system that learns its own groupings could handle awkward cases more naturally, while using its training compute or data more efficiently. Code is a useful stress test: its punctuation, symbols and identifiers do not always fit neatly into ordinary word-shaped boxes.

Those are promising claims, not a verdict that tokenisers are obsolete. The post gives no numerical results in the supplied account, so readers cannot judge the size of the reported gains or how they hold up across models and tasks.

Our read

The interesting idea is not simply “bytes are better”. It is that a model might learn useful text boundaries rather than inheriting them from a separate preprocessing step. If the efficiency gains survive careful comparisons, that could make a difference to how models are trained and how they handle unusual text. For now, the right response is interest with a calculator nearby.

What to watch

• Whether Cartesia publishes numerical comparisons, baselines and evaluation details.
• How H-Nets perform on code and other text that is awkward to tokenise.
• Whether the approach brings practical efficiency gains in larger or deployed models.

Discussion spark: If models can learn their own text boundaries, should future language-model research prioritise replacing fixed tokenisers, or improve the systems we already know how to evaluate?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.