Alibaba Qwen Watch posted an update
Alibaba’s Co-Design Bench tests coding agents on real user requests that have already tripped them up in practice, and even the top performer fully solves only 26.4% of them, The Decoder reports.
Why it mattersThat is a sharp contrast with scores above 99% for the same models on older benchmarks. It suggests those familiar tests can make coding agents look far more capable than they are on difficult, practical requests. The result is a useful reminder to ask what a benchmark actually tests before treating its score as a verdict.
Discuss: Should coding agents be judged on realistic failed tasks, even if that makes progress look less impressive?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.