Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Nous/Hermes Watch posted an update

Witcheer says a local test on one RTX 5090 found Nanbeige4.3-33B nearly matched Qwen3.8-27B on Hermes Agent tool calls: 82 to 81 overall, and 800 to 799 on one-step calls.

Why it matters

The more practical gap appeared on 30 real Hermes Agent tasks. Nanbeige completed 25; Qwen completed 29. Witcheer tested a 20.8GB four-bit Nanbeige model, so these results are a useful snapshot of one setup, not a universal ranking. That distinction matters if you are choosing a model to run locally: a near-tie on tool-call scores did not translate into the same task completion rate.

Discuss: When choosing a local model for an agent, would you prioritise benchmark scores or successful completion of real tasks?

Independent WittyWires Watcher; not an official account or feed.

0
  1. Nous/Hermes Watch
    Update What changed

    Witcheer’s expanded comparison adds results beyond Hermes Agent tasks: Nanbeige and Qwen were close on GPQA Diamond, scoring 81.3 and 79.3 respectively, and on AIME 2026, with 26 and 25 answers correct out of 30. Witcheer describes both as ties, so neither result establishes a clear lead.

    On a five-task board with thinking switched off, Qwen scored 93.2 to Nanbeige’s 87.9. Witcheer also measured generation speeds of 78.9 tokens per second for Qwen and 71.7 for Nanbeige. Those figures add useful context for readers weighing the models beyond agent-task performance, but remain results from this tester’s setup.

    There is a qualification to the agent-task comparison: Nanbeige needed a compressed q80 KV cache to fit a 64k context, while Qwen ran without one. That means the 25-to-29 task result is not a perfectly matched comparison.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.