SWE-Bench Pro V2 raises the bar for testing AI coding agents

Scale Labs and Reflection have released SWE-Bench Pro V2, a tougher and more tightly controlled benchmark for AI software-engineering agents. It covers 642 tasks across 11 repositories, including a new 51-task Hard split, and is designed to make claims about coding ability harder to inflate with leaky tests, broken tasks or a conveniently open internet.

Watch Desk analysis

What happened

The new benchmark uses a locked evaluation protocol, removes invalid tasks, corrects task instructions and improves dependency support. Web tools are disabled during the agent phase, while evaluation uses pristine images, stricter re-grading and extensive human expert review, according to the SWE-Bench Pro V2 release.

The central change is not simply more questions. Scale Labs and Reflection say the refresh is intended to keep pace with increasingly capable frontier models and provide a more rigorous measure of how well software agents solve real repository-level tasks. The new 51-task Hard split should provide a sharper stress test than the broader public set, although the supplied release does not establish how individual systems perform on it.

Key findings

• 642 tasks across 11 repositories
  The refreshed public split gives developers a substantial, defined workload for comparing software agents.
• A 51-task Hard split
  The new subset is intended to make difficult agent performance easier to distinguish.
• Network-locked evaluation
  Disabling web tools during the agent phase reduces the chance that outside browsing quietly supplies the answer.
• Pristine images and human review
  Fresh environments and stricter re-grading aim to catch fragile solutions and questionable task labels.
• Invalid tasks removed
  Cleaning the dataset matters because a benchmark cannot fairly measure agents if the questions themselves are broken.

Why it matters

AI coding scores are increasingly treated as shorthand for whether an agent can do useful engineering work. That makes benchmark hygiene rather less academic than it sounds. If an agent can browse for answers, exploit a faulty task or benefit from a stale environment, a headline score may measure the test harness as much as the model.

SWE-Bench Pro V2 gives teams a more defensible basis for comparison, but it is still a benchmark, not a probation report from a software team. A stronger score should mean more when the evaluation is harder to game, yet it cannot by itself settle questions about maintainability, security, collaboration or the messy requirements that arrive without a neat repository boundary.

Our read

This is a meaningful infrastructure update for AI coding evaluation, not just another leaderboard refresh. The locked protocol, disabled web tools and human review are the useful story. The benchmark’s credibility will now depend on whether results are reproducible, whether the Hard split separates strong systems cleanly and whether developers can inspect enough of the methodology to understand what a score actually means.

For readers comparing coding agents, treat future scores on V2 as evidence rather than verdicts. A model that tops a benchmark may still produce code that requires a patient colleague, a test suite and possibly a small exorcism.

What to watch

• Independent results:
  whether multiple teams publish results under the same protocol.
• Hard-split separation:
  whether the 51-task subset meaningfully distinguishes leading agents.
• Reproducibility:
  whether reported scores survive fresh environments and repeated runs.
• Real-world correlation:
  whether V2 performance tracks useful engineering outcomes outside the benchmark.

Discussion spark: Should AI coding agents be judged mainly by locked benchmark performance, or should evaluation prioritise longer-term code quality and maintenance in real projects?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.