A new benchmark finds that even advanced computer-use agents can reach the wrong result by taking the wrong route. OSWorld-Pro evaluates more than 300 tasks and 2,800 subgoals, exposing the small, consequential mistakes hidden by a tidy final answer.
Watch Desk analysis
What happened
Researchers introduced OSWorld-Pro to assess computer-use agents through the process of completing tasks, rather than judging only the final state. The benchmark uses human annotations and human-aligned language-model judges to examine whether an agent’s actions remain relevant to the subgoal in front of it.
The researchers report that Claude Opus 5 achieved 75.7% on the benchmark. The evaluation highlights failure modes including actions unrelated to the current subgoal and mistakes made while clicking through interfaces, according to the researchers’ paper.
Key findings
- Process matters
An agent can eventually produce a plausible outcome while wasting steps or drifting away from the task. - Clicking remains a failure point
Interface actions can go wrong even when the agent appears to understand the broader instruction. - The benchmark is granular
More than 300 tasks are broken into 2,800 subgoals, giving evaluators more places to identify where performance slips. - 75.7% is not a clean bill of health
The reported score leaves plenty of room for failures in ordinary computer-use workflows.
Why it matters
A final-answer score can make an agent look more reliable than it is. In a real workplace, an irrelevant click may alter a record, skip a check or send the task down an expensive side road before anyone notices. Process-based evaluation makes those weak points visible.
That gives developers something more useful than a single leaderboard number. They can inspect whether an agent misunderstood the instruction, chose an unnecessary action or simply clicked the wrong target. The benchmark does not magically fix those problems, but it makes them harder to hide behind a successful-looking outcome.
Our read
This is the sort of benchmark the agent industry needs more of. “Did it finish?” is a useful question, but “what on earth did it do along the way?” is often the one that keeps the incident report from writing itself.
The practical takeaway is to ask vendors and internal teams for step-level evidence before trusting agents with sensitive interfaces. A polished completion is not the same thing as a dependable process, and OSWorld-Pro offers a sharper way to tell the difference.
What to watch
- Whether future models improve on process errors rather than only final-task scores.
- Whether independent evaluations reproduce the reported 75.7% result.
- Whether benchmark failures predict mistakes in real applications and workflows.
- Whether developers expose action traces to users and auditors by default.
Discussion spark: Should an AI agent be considered successful if it reaches the right result through unsafe or inexplicable steps, or should process quality be a hard requirement?
Sources and evidence
- OSWorld-Pro: Process-based Evaluation for Computer Use Agents (22 September 2026, 04:17 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.