AWS launches CloudWatch Omni to watch AI agents before they wander into trouble

AWS has launched CloudWatch Omni, a new observability service for applications and AI agents that brings tracing, evaluation and investigation into one shared workspace. The important bit is not another dashboard. Omni is designed to show developers and operators what an agent actually did, where it went wrong and whether a change quietly made its answers worse.

AWS AI Watch analysis

What happened

According to AWS, CloudWatch Omni is available now through a dedicated web experience, with access managed through enterprise single sign-on rather than AWS Console credentials. Existing CloudWatch customers can try it from the CloudWatch console, and the service is built on OpenTelemetry, so telemetry already sent to CloudWatch can appear without reconfiguring the existing setup.

For AI agents, the service offers a VS Code and Kiro extension, a standalone operations view, structured traces of model calls and tool invocations, and a playground for comparing prompts and configurations. AWS says teams can build test datasets from production traffic, run experiments and detect regressions automatically. It also includes 17 built-in evaluators covering measures such as coherence, helpfulness, faithfulness and routing correctness.

Key findings

• Trace Explorer
  A structured timeline shows model calls, tool invocations, reasoning steps, inputs, outputs, token use and latency.
• Built-in evaluation
  Teams can score agent responses against quality dimensions rather than relying only on latency and error rates.
• Prompt and model comparison
  The Playground and Experiments views compare configurations before a change reaches production.
• Local-first development
  Developers can use Omni locally, then optionally connect to AWS for persistent telemetry, shared traces and production dashboards.
• One investigation space
  Operators can monitor a fleet in a separate web experience, while developers inspect the same traces inside their IDE.

Why it matters

Traditional monitoring can tell a team that a request failed or became slower. Agent systems need a more awkward question answered: why did the system choose that tool, prompt or next step? A response can look technically healthy while becoming less useful, less faithful or more expensive.

Omni is AWS’s attempt to turn that behaviour into something inspectable. The practical payoff is a way to compare a new prompt against the old one, replay a dataset and look for quality regressions before users discover them the entertaining way. It also makes the hand-off between developer and operator less dependent on screenshots and heroic archaeology in log files.

Our read

This is a meaningful piece of agent infrastructure, not merely a fresh coat of dashboard paint. The strongest feature is the link between traces and evaluation: seeing an agent call the wrong tool is useful, but measuring whether a change makes that mistake more common is better.

There is still an important boundary. AWS is describing its own product and capabilities, not publishing independent evidence that the evaluators reliably judge every agent or that the service will replace careful human review. The sensible starting point is to use Omni as an evidence trail and test harness, not as an oracle wearing an SSO badge.

What to watch

• Whether teams can export traces and evaluation data cleanly across cloud providers.
• How well the built-in evaluators agree with expert human assessment on difficult tasks.
• The cost and retention rules for production traces, especially those containing sensitive prompts or tool outputs.
• Whether OpenTelemetry support makes Omni genuinely portable or simply easier to enter than to leave.

Discussion spark: Should agent observability become a standard independent layer across cloud providers, or is a tightly integrated platform like Omni more useful even if it increases lock-in?

not affiliated with or endorsed by Amazon Web Services (AWS)