Watch Desk posted an update
A developer’s test puts a compact, quantised Bonsai model close to a much larger Qwen model on a set of maths problems. The reported scores were 46 and 47 out of 63, respectively.
Why it mattersThe Bonsai version used 7.2GB of storage, against 22.9GB for the Qwen version. That is a striking size difference, though this single test is not a general verdict on either model. The results were posted by witcheer, who says the test used this year’s AIME and HMMT questions and a 32,000-token thinking budget. Which matters more for local models: matching top-line scores, or doing it with far less memory?
Discuss: For local models, which matters more: matching top-line scores, or doing it with far less memory?
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk
Watch Desk Update What changedIn the developer’s comparison, the compact Bonsai model was correct on 95.4% of answers that finished, against 96.1% for the larger Qwen model. The figures add a useful qualification to the near-match: the scores were close, but not identical.
The models also diverged on individual problems. Bonsai solved six questions that Qwen missed, while Qwen solved seven that Bonsai missed. Their similar totals therefore conceal different strengths on this set.
Those figures come from witcheer’s test of 63 questions drawn from this year’s AIME and HMMT, using a 32,000-token thinking budget. They are results from one developer-run comparison, not a general measure of either model’s ability.
Sources and evidence
- round 2 for the Bonsai files: this year's AIME and HMMT, 63 problems, 32k thinking budget, my RTX 5090. >Ternary-Bonsai-2-27B PQ20, 7.2 GB: 46 >Qwen3.8-27B Q6K, 22.9 GB: 47 >Ternar: Witcheer reports that, among completed answers in a 63-question AIME and HMMT test, the Bonsai model was correct 95.4% of the time and Qwen 96.1%; Bonsai solved six questions Qwen missed, while Qwen solved seven Bonsai missed.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.