Dylan McClung has pulled the chat templates from 102 of the most-downloaded conversational models on Hugging Face and pushed the same conversation through every one of them. The result: 57 distinct templates, 52 distinct outputs and 40 models that quietly add a system prompt nobody wrote. Nothing crashed in the making of this survey, which is rather the point. The chat template is a Jinja document that runs on every request, ships inside the repo with thin documentation and can degrade an agent without ever raising an error. It is the least-examined part of a model release, and this survey shows what that neglect costs.
Watch Desk analysis
What happened
McClung published his chat template archaeology as a Hugging Face Community Article on 16 September, with the script and full dataset alongside. Models were listed by download count, quantisations and precision re-uploads dropped, and each template pulled from chattemplate.jinja, tokenizerconfig.json or chattemplate.json, whichever existed.
Then he did the part that matters: every template was parsed and rendered against a fixed probe conversation containing a system message, a tool call, a tool response and a prior assistant turn with a reasoning block. Rendering is where the survey earns its keep. A regex for reasoning-stripping logic matched 47 templates; only 10 of them actually stripped anything. Static analysis tells you what an author typed, not what a model receives, and the gap between the two is where these bugs live.
Key findings
- Tool-call arguments have no agreed type
18 templates accept only a dict, 4 only a JSON string; pick the wrong one and Jinja throws a TypeError before generation even starts. - 40 of 102 write a system prompt you never asked for
Qwen2.5 introduces itself, Llama 3.x injects a knowledge cutoff and today's date, and gpt-oss builds a whole preamble. - 10 templates delete reasoning blocks from history
Six DeepSeek R1 repos, three NVIDIA Nemotron repos and one more silently erase the model's own earlier thinking on multi-turn replay. - There is a hidden keyword API
48 templates read undocumented arguments across 20 distinct names, and four vendors invented four labels for the same thinking toggle, two of them polarity-inverted. - Hermes XML has won the tool-calling dialect war
52 templates use it, largely because Qwen does and Qwen derivatives dominate Hugging Face downloads. - Lineage goes unrecorded
Only 41 of 102 repos declare a base model and exactly one declares its relationship, so a finetune inherits its parent's quirks with no pointer back. - Templates grow without a standard
Llama 3 to 3.1 was a 12× jump in one release, and gpt-oss-20b now ships 16,714 characters of Jinja that run before the model sees a token.
Why it matters
Agent frameworks assume a stable interface: OpenAI-shaped messages, a tools array, a system prompt. The chat template is where that assumption meets per-model, per-release variability with no changelog. Miss a named tooluse template and a tool-trained model receives a prompt that never mentions tools, as McClung showed with the default template on Hermes-3-Llama-3.1-8B. Nothing crashes, but the output quietly gets worse, which is the hardest kind of bug to notice.
The benchmarking implication may sting more. If 40 of the most-downloaded models inject their own system prompts, then "no system prompt" comparisons are measuring something that never existed. Anyone running evals is advised to render the template with an empty system slot and read what actually comes out.
Our read
Read the Jinja before you trust the README. McClung's checklist is short and worth more than most model cards: check whether a named tooluse template exists before assuming the default supports tools, test both argument types against your specific model, check whether reasoning blocks survive replay, and read the isdefined guards to learn the model's undocumented settings. None of this is malice. It is accretion without a standard, and the people best placed to fix it are the ones who read what actually ships.
What to watch
- Standardisation pressure.
Whether transformers or major model publishers respond with documented template contracts. - The dataset.
Whether the survey repeats with new releases and becomes a running index. - Metadata adoption.
Whether more repos start declaring basemodel lineage after this. - Dialect drift.
Whether tool-calling converges further on Hermes XML or fragments again.
Discussion spark: With 40 of the 102 most-downloaded models quietly writing their own system prompt, should template behaviour be a documented, testable part of every model release, or is dodging the chaos simply part of every framework builder's job?
Sources and evidence
- Chat Template Archaeology: What the Top Instruct Models Actually Do to Your Prompt (16 September 2026, 21:30 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.