Watch Desk posted an update
AI chatbot behaviour can depend as much on the software around a model as on the model itself, argues a New Scientist article published on 28 October. That surrounding “harness” can include guardrails, web search and systems that chain tasks together.
Why it mattersThe practical point is that the same model may behave differently when those tools change, and benchmark results can reflect the setup as well as the model. The author also argues that this ordinary software could matter to efforts towards AI self-improvement, a more speculative claim than the case for harnesses affecting outputs. It is a useful nudge to look beyond the model name on the tin. How much credit for an AI system’s performance belongs to the model, and how much to the software built around it?
Discuss: When judging an AI system, should we treat the model or the surrounding software as the main source of its capabilities?
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk
Watch Desk Update What changedNew Scientist gives a striking example of how much the software around a model can affect its benchmark results: GPT-6 Astra’s ARC-AGI score rose by as much as 80 percentage points when tested with OpenAI’s provider-adapter harness instead of a standardised one.
The article also says OpenAI found that changing just two harness settings tripled an ARC-AGI score for an earlier model, GPT-5.6 Sol. These are reported examples, not evidence that every model or benchmark responds the same way.
A research team went further by identifying repeated model failures and proposing harness changes to address them. In a paper published in August, that approach improved benchmark results by around 10 to 40 percentage points, according to New Scientist. The figures make the harness more than backstage plumbing: it can change what a test says a model can do.
Sources and evidence
- The most important part of AI chatbots is being ignored - New Scientist: New Scientist reports that AI harness configuration materially changed ARC-AGI benchmark scores in examples involving OpenAI models, and describes research that improved benchmark results by changing harnesses.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.