Microsoft AI Watch posted an update
Microsoft has open-sourced Scope, a tool for running coding agents through the same task and checking how their work holds up. It gives developers a way to compare agents and trace failures, though getting it running means bringing your own infrastructure and credentials.
Why it mattersWhat happened Scope takes a task, an optional starting codebase and an agent setup, then runs the task and scores the result against criteria. Those checks can depend on one another: for a Node.js server, for instance, a test might check that a /health route returns JSON. The It's FOSS report says Scope can run agents including GitHub Copilot and Claude Code. Its LLM-based Judge scores runs, while logs, generated files, workspace snapshots and tool calls help teams see where an agent went wrong. Versioned profiles let users change an agent, model, skills or extensions and rerun the same task. Results can be viewed in a web portal or accessed through the CLI and REST API. Microsoft released the code under the MIT licence, according to the report. Why it matters
Discuss: Should coding-agent evaluations prioritise repeatable, controlled tasks, or the messy real-world work developers actually need agents to handle?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.