Changing the software harness around an AI model produced a wide spread in IBM’s test of business-app tasks: completion ranged from 30% to 81%. The result is a useful reminder that choosing an agent is not just choosing its model, though the comparison has important limits.
Watch Desk analysis
What happened
IBM’s CUGA team compared CUGA, Hermes and LangChain’s Deep Agents using the same open-weight GLM-5.3-flash model on 90 AppWorld tasks across 30 scenarios. The team reports task completion rates ranging from 30% to 81%; on its measure of scenarios where all three variants passed, CUGA scored 20, Hermes eight and Deep Agents two.
The IBM Research comparison says each task had one attempt. AppWorld tests multi-step work across nine apps and 457 APIs, but the researchers describe it as a proxy for some enterprise automation skills, not a direct test of business deployments. They also note that the harness setups were not identical and that the runs did not test policy enforcement.
Why it matters
A model may be capable of a task and still stumble when the surrounding agent cannot reliably navigate tools, APIs and multi-step instructions. For teams evaluating agents, this makes the harness and its fit for the job part of the buying decision, not decorative plumbing.
The results are not a universal league table. They come from one benchmark, a small task set and a comparison run by the CUGA team. The study does, however, put a number on how much the surrounding software may matter in this particular test.
Our read
The most useful takeaway is the size of the gap, not a trophy for one harness. IBM’s test gives teams a reason to compare complete agent setups on work they actually need done. It does not establish that CUGA will outperform the others in every workload, or that any system is ready to handle sensitive business actions without safeguards.
What to watch
- Whether the comparison is repeated on more tasks and other benchmarks.
- Whether independent teams can reproduce the results with matched configurations.
- How the harnesses perform when policy enforcement and approvals are tested.
- Whether the same differences appear in real enterprise workflows.
Discussion spark: When evaluating an AI agent, should organisations prioritise the model’s benchmark scores or test the entire harness on their own workflows?
Sources and evidence
- Same Model, Different Harness: CUGA, Hermes, and Deep Agents – When to Pick CUGA? (3 November 2026, 15:57 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.