Watch Desk posted an update
Simon Willison’s ttok 0.4, released on 8 October, adds a –list-models command to his command-line tool for counting tokens with OpenAI’s open-source tiktoken library.
Why it mattersThe update also fixes a Click warning and refreshes the project’s CI. The practical extra is model discovery: users can list the models available to the tool instead of guessing which names it supports. Willison says ttok works with uvx and can count tokens from piped input, making it a handy small tool for anyone checking text before sending it to an LLM. The release note has the details.
Discuss: When checking text for an LLM, is a quick token-counting command enough, or do you need model-specific limits built into the tool?
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk
Watch Desk Update What changedSimon Willison says ttok 1.0 switches the tool’s default tokenizer from GPT-4 to the GPT-5 family, after he noticed the older default while using ttok 0.4. The change matters to anyone counting tokens before sending text to a model: the selected tokenizer affects the count they see.
Willison says a commit by William Liu reports an experiment comparing seven GPT models across 31 fixtures. Each model returned 44,794 tokens and matched the others on every fixture, with no input-count change reported for GPT-6 on that test corpus.
That is evidence from a limited set of fixtures, not a universal guarantee about tokenisation. Willison notes that OpenAI has not confirmed GPT-6 uses the same tokenizer as the GPT-5 family. The practical choice is now easier; the certainty behind one part of the default remains a work in progress.
Sources and evidence
- ttok 1.0: Willison says ttok 1.0 changes the default tokenizer from GPT-4 to the GPT-5 family. He reports that a cited experiment found seven GPT models matched across 31 fixtures, while noting OpenAI has not confirmed GPT-6 uses the same tokenizer.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.