The software wrapped around an AI model can change what it can do, and how well it performs. A New Scientist article argues that this often-overlooked “harness” may matter as much as the model itself.
Watch Desk analysis
What happened
A harness is the software connecting a model to its interface, tools and guard rails. It can let a chatbot search the web, read documents or work through a problem in repeated steps. More capable harnesses, including those that let coding models control a computer, give the underlying model a different job to do.
New Scientist’s Jacob Aron illustrates the effect by running the same GPT-6 model in different setups. He says its answer to “Who am I?” changed between the API, an unlogged ChatGPT session and his personal account, which had information about him. The article also says GPT-6 Astra’s score on the ARC-AGI test rose by as much as 80 percentage points with OpenAI’s provider adapter, compared with a standardised harness. For an earlier model, the article says OpenAI found that changing two harness settings tripled its score.
Why it matters
A model’s headline benchmark score may say less than it appears to if tools and settings differ between tests. And when someone chooses an AI service, they are choosing more than a model: the surrounding software can shape its answers, capabilities and access to information.
That makes the harness a practical target for developers and researchers, not just an obscure layer for software engineers. The article also describes research testing whether models can help improve their own harnesses, with one paper reporting benchmark gains of around 10 to 40 percentage points after researchers identified repeated failures and proposed changes.
Our read
The model gets the marquee billing; the harness may be doing much of the stagecraft. That is a useful reason to ask what tools and settings were used before treating one model ranking as the final word. The scores and examples here are reported by New Scientist, and results from particular test setups are not a universal measure of real-world usefulness.
What to watch
- Whether future model evaluations make harnesses and tool access easier to compare.
- Whether AI companies publish more detail about the software around their models.
- Whether research on models improving their own harnesses produces repeatable gains.
Discussion spark: When you compare AI models, should a standardised harness matter more than the tools each provider has built around its model?
Sources and evidence
- The most important part of AI chatbots is being ignored (28 October 2026, 09:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.