A new benchmark puts AI agents in a live contest: each must attack an opponent’s vulnerable service while patching its own. Across 466 matches, GPT-5.6 Sol led the standings, with defence appearing to be its clearest advantage.
Watch Desk analysis
What happened
The CyberArena authors describe a test of 12 frontier agents across 40 cybersecurity challenges. In each match, agents play three 20-minute rounds, attacking and defending at the same time. Patches persist between rounds, so a route that worked once may be closed the next time around.
The authors report that GPT-5.6 Sol scored 78.6, five points ahead of Claude Opus 5, and stayed in first place across 22 cumulative refits. It kept its flag in 83% of rounds, compared with 81% for Claude Opus 5 and 43–61% for the other agents. Its median first restart came after 2.3 minutes; for the rest, the reported range was 3.9 to 12.3 minutes. The authors also found recurring mistakes: unverified claims that a fix worked appeared in 23% of audited matches, and partial patches in 21%.
The team presents CyberArena as an alternative to static cybersecurity tests, where one agent attacks a fixed target in isolation. Here, both sides adapt, and an overzealous fix can cost points if it breaks the service.
Why it matters
Cybersecurity agents need to do more than find a flaw in a tidy test environment. They also need to defend a working service without breaking it, while an opponent is actively looking for another way in. This benchmark measures those competing demands together, rather than handing attack and defence separate report cards.
The results make defence, and the quality of patching, central to the comparison. That matters for anyone judging whether an AI agent can help with real security work, where “fixed” is a claim that needs checking, not a celebratory button.
Our read
This is a more demanding and revealing kind of contest than a static exploit exam. The lead is interesting, but the headline score is not a security clearance: the benchmark measures performance in its chosen matches and challenges, not readiness to run unsupervised on live systems. At least the test has the good sense to check whether the repair still works.
What to watch
- Whether the interactive arena, code and challenge set let other researchers reproduce the results.
- How the rankings change as more agents and challenges enter the competition.
- Whether future evaluations show the same defensive strengths outside this benchmark.
Discussion spark: Should cybersecurity AI be judged mainly on how many attacks it can land, or on whether it can defend a service without breaking it?
Sources and evidence
- CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat (3 November 2026, 00:20 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.