Weights & Biases says its Arya research agent can turn errors from real production runs into test tasks, then use those tasks to evaluate and improve new versions of its own code. The approach links real-world failures to repeatable offline testing, a useful engineering loop with a very human catch: someone still has to check whether the scores translate back to live behaviour.
Watch Desk analysis
What happened
At a demonstration described by StartupHub.ai on 28 September, Weights & Biases’ Zubin Aysola showed Arya working with production traces recorded in Weave. The system can convert a production error into a regression task, run evaluations against candidate versions and compare the results. The company’s task library contains 886 YAML-defined tasks, and the reported scores hover around 66% on tracked task sets.
StartupHub.ai says production code is mirrored into a research environment every four hours. The demonstration also showed nightly jobs testing candidate versions. These are reported company figures and a demonstration, not independent evidence that improvements reliably carry over to production.
Why it matters
The practical idea is to let an agent learn from the mistakes it actually makes, rather than relying only on a tidy benchmark chosen in advance. Turning a missed software call into a repeatable test could help teams spot regressions and compare changes against real tasks.
But an offline score is not a warranty for live performance. The report notes that simulations are costly and take time, and that people still need to review the gap between test results and real behaviour. The feedback loop is promising; the unglamorous evaluation work remains very much employed.
Our read
This is a concrete example of self-improvement as an engineering process: capture traces, make them testable, then evaluate candidate changes. The strongest takeaway is not that Arya can rewrite itself, but that production failures can become part of the test suite. Teams considering a similar setup should pay close attention to how tasks are scored and how candidates are checked against live use.
What to watch
- Whether the 886-task library grows and how teams decide which tasks reflect useful product behaviour.
- Whether offline evaluation scores predict results in live workflows.
- How much the simulation and human review process costs as the task library expands.
Discussion spark: Should AI agents be allowed to propose changes to their own code when every change is tested against production-derived tasks, or should human-written tests remain the final gate?
Sources and evidence
- Weights & Biases made its agent improve itself – startuphub.ai (28 September 2026, 09:08 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.