Discussion

CWE-bench v2 puts AI coding agents through 100 real-world security tasks

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#10986

Collinear AI has published CWE-bench v2, a benchmark of 100 tasks testing whether AI agents can find and fix security weaknesses in real open-source repositories. Even the leading model in the reported results fails nearly one in five tasks on every attempt, a useful reminder that a strong score is not a security forcefield.

Watch Desk analysis

What happened

The benchmark gives an agent a repository checkout and a short instruction: find and fix a vulnerability without breaking existing functionality. Its 100 tasks cover 91 weakness classes, and the project says the set, harness and deterministic grader are public. Tasks are also hand-reviewed, with checks designed to reject fixes that break the original software or merely reproduce a known patch. A separate held-out set is available for tracking progress.

Collinear AI’s post on Hugging Face reports that Gemini 4 Argon leads 11 evaluated models at 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows closely at 67.3% and 78.0%. Across the set, 95 tasks were solved by at least one model, but the leader failed 19% of tasks on every attempt.

The results also expose a cost trade-off. DeepSeek V4.1 Flash reached 57.0% pass@1 for $0.04 per rollout; Opus added ten percentage points at $1.83, while Gemini edged ahead at $3.92. The dataset is gated, so users must accept its access conditions before running it.

Why it matters

A coding agent that can repair a security flaw without damaging surrounding software is more useful than one that merely spots a familiar pattern. CWE-bench tests both sides of that job, and its broad set of weakness classes makes the results harder to reduce to a single party trick.

The price spread matters too: teams weighing automated security work are choosing between performance and the cost of repeated attempts. But these are benchmark results, not evidence that an agent will reliably secure a live production system.

Our read

This is a welcome move from polished demos towards reproducible tests with real code and a clear pass condition. The most interesting result is not just who leads, but how much failure remains and what each extra point costs. Security teams should treat the benchmark as a comparison tool, not a substitute for human review or testing in their own repositories.

What to watch

  • Whether more models are evaluated against the public tasks and the held-out set.
  • Whether results hold up across different repositories and vulnerability classes.
  • Whether lower-cost models can close the gap without sacrificing reliable fixes.

Discussion spark: For AI-assisted security work, would you choose the highest pass rate available, or a cheaper model that leaves more fixes for human review?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.