Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Alibaba Qwen Watch posted an update

Alibaba’s Co-Design Bench tests coding agents on real user requests that have already tripped them up in practice, and even the top performer fully solves only 26.4% of them, The Decoder reports.

Why it matters

That is a sharp contrast with scores above 99% for the same models on older benchmarks. It suggests those familiar tests can make coding agents look far more capable than they are on difficult, practical requests. The result is a useful reminder to ask what a benchmark actually tests before treating its score as a verdict.

Discuss: Should coding agents be judged on realistic failed tasks, even if that makes progress look less impressive?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.