Vera says AI agents pass only half its cloud-operations benchmark

AI agents built for coding still stumble when asked to manage live cloud infrastructure, according to research shared by Vera Platforms with The Register. Its new CloudOpsBench is designed to test those operational tasks in a simulator, and Vera CEO Ang Chen says leading models score about 50 per cent.

Watch Desk analysis

What happened

The benchmark tests jobs such as deleting decommissioned S3 buckets, repairing a compliance-event pipeline and restoring DynamoDB write autoscaling. Chen says the tasks were written and peer-reviewed by experienced infrastructure operators, and that the simulator reproduces Amazon’s behaviour closely enough for its results to match those of the corresponding cloud APIs.

Vera, founded by University of Michigan researchers, is building three related products: vera-gym, a simulated infrastructure environment; vera-bench, the benchmark; and vera-agents, agents trained in the environment. Chen told The Register that vera-agents are expected in the first quarter of 2027.

Why it matters

Cloud operations leave less room for a confident guess than code generation does. A mistaken infrastructure change can interrupt a service, expose a security weakness or delete data. Chen gave the example of Claude and OpenAI’s Codex being asked to decommission AWS resources but overlooking resources outside the default region, leaving the job only partly done.

A benchmark that tests those chores in a simulator could help teams find failures before handing an agent access to production systems. But the roughly 50 per cent pass rate and the simulator’s claimed equivalence come from Vera’s account; they are not independent evaluations of the benchmark or its results.

Our read

This is a useful reality check for anyone tempted to treat fluent code generation as operational competence. The interesting question is not whether an agent can perform a cloud task once, but whether it notices the awkward details, handles the full environment and does so reliably. Vera’s benchmark makes that harder question concrete, though its own agents will need to earn their credentials outside the pitch deck.

What to watch

• Whether Vera publishes CloudOpsBench tasks, scoring details and results that others can reproduce.
• How performance compares across models and against experienced operators.
• Whether vera-agents arrive as planned in the first quarter of 2027, and what evidence supports their readiness.

Discussion spark: Would you let an AI agent make changes to live cloud infrastructure after it passed a benchmark, or should a human remain responsible for every production change?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.