Collinear AI has published CWE-bench v2, a benchmark of 100 tasks testing whether AI agents can find and fix security weaknesses in real open-source repositories. Even the leading model in the reported results fails nearly one in five tasks on every attempt, a useful reminder that a strong score is not a security forcefield.
Watch Desk analysis
What happened
The benchmark gives an agent a repository checkout and a short instruction: find and fix a vulnerability without breaking existing functionality. Its 100 tasks cover 91 weakness classes, and the project says the set, harness and deterministic grader are public. Tasks are also hand-reviewed, with checks designed to reject fixes that break the original software or merely reproduce a known patch. A separate held-out set is available for tracking progress.
Collinear AI’s post on Hugging Face reports that Gemini 4 Argon leads 11 evaluated models at 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows closely at 67.3% and 78.0%. Across the set, 95 tasks were solved by at least one model, but the leader failed 19% of tasks on every attempt.
The results also expose a cost trade-off. DeepSeek V4.1 Flash reached 57.0% pass@1 for $0.04 per rollout; Opus added ten percentage points at $1.83, while Gemini edged ahead at $3.92. The dataset is gated, so users must accept its access conditions before running it.
Why it matters
A coding agent that can repair a security flaw without damaging surrounding software is more useful than one that merely spots a familiar pattern. CWE-bench tests both sides of that job, and its broad set of weakness classes makes the results harder to reduce to a single party trick.
The price spread matters too: teams weighing automated security work are choosing between performance and the cost of repeated attempts. But these are benchmark results, not evidence that an agent will reliably secure a live production system.
Our read
This is a welcome move from polished demos towards reproducible tests with real code and a clear pass condition. The most interesting result is not just who leads, but how much failure remains and what each extra point costs. Security teams should treat the benchmark as a comparison tool, not a substitute for human review or testing in their own repositories.
What to watch
- Whether more models are evaluated against the public tasks and the held-out set.
- Whether results hold up across different repositories and vulnerability classes.
- Whether lower-cost models can close the gap without sacrificing reliable fixes.
Discussion spark: For AI-assisted security work, would you choose the highest pass rate available, or a cheaper model that leaves more fixes for human review?
Sources and evidence
- CWE-bench v2: More coverage, quality, and transparency (3 November 2026, 00:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.