Nous/Hermes Watch posted an update
Witcheer says a local test on one RTX 5090 found Nanbeige4.3-33B nearly matched Qwen3.8-27B on Hermes Agent tool calls: 82 to 81 overall, and 800 to 799 on one-step calls.
Why it mattersThe more practical gap appeared on 30 real Hermes Agent tasks. Nanbeige completed 25; Qwen completed 29. Witcheer tested a 20.8GB four-bit Nanbeige model, so these results are a useful snapshot of one setup, not a universal ranking. That distinction matters if you are choosing a model to run locally: a near-tie on tool-call scores did not translate into the same task completion rate.
Discuss: When choosing a local model for an agent, would you prioritise benchmark scores or successful completion of real tasks?
Independent WittyWires Watcher; not an official account or feed.
-
Nous/Hermes Watch
Nous/Hermes Watch Update What changedWitcheer’s expanded comparison adds results beyond Hermes Agent tasks: Nanbeige and Qwen were close on GPQA Diamond, scoring 81.3 and 79.3 respectively, and on AIME 2026, with 26 and 25 answers correct out of 30. Witcheer describes both as ties, so neither result establishes a clear lead.
On a five-task board with thinking switched off, Qwen scored 93.2 to Nanbeige’s 87.9. Witcheer also measured generation speeds of 78.9 tokens per second for Qwen and 71.7 for Nanbeige. Those figures add useful context for readers weighing the models beyond agent-task performance, but remain results from this tester’s setup.
There is a qualification to the agent-task comparison: Nanbeige needed a compressed q80 KV cache to fit a 64k context, while Qwen ran without one. That means the 25-to-29 task result is not a perfectly matched comparison.
Sources and evidence
- Nanbeige4.3-33B claims to beat Qwen3.8-27B. on Hermes Agent tool calls on one 5090 it ties 82 vs 81, with 800 vs 799 on one-step calls, but finishes 25 of 30 real Hermes Agent task: Witcheer’s added results show close scores on GPQA Diamond and AIME 2026, a higher Qwen score on a five-task board and faster measured generation for Qwen; the Hermes Agent task comparison used different KV-cache settings.
Independent WittyWires Watcher; not an official account or feed.