Discussion

Reflection’s vending-machine AI agent passed jailbreaks, but trusted a fabricated warning

In The Watch Desk

Reflection AI Watch
Reflection AI WatchParticipantOpening post
#15098

Reflection says its AI agent Beam resisted a series of attempts to break its rules while running a real vending business. But the company’s own evaluation also found it could carry a fabricated claim into its memory and let that claim shape its behaviour for days. That is a useful distinction: standing firm against an obvious jailbreak is not the same as reliably handling a messy, multi-step job.

Reflection AI Watch analysis

What happened

Reflection and Andon Labs set up AI agents to run vending businesses in their New York and San Francisco offices. The agents could choose stock and prices, order supplies, handle customer requests and complaints, and work within budgets and safety rules. Staff interacted with them through Slack, where they also tested their limits.

In Reflection’s account of the evaluation, Mr. Beam rejected sustained attempts to persuade it to sell nicotine pouches, including forged messages and a fake rival agent. But it also accepted an employee’s false claim that the head of the safety team had been fired. Reflection says that information entered the agent’s long-term memory and influenced its behaviour for several days. The company also describes the agent treating a fabricated report of illness after drinking expired soda as genuine.

Why it matters

The evaluation puts a practical wrinkle in the usual jailbreak scorecard. An agent can refuse direct requests to break a rule and still be misled by bogus information, carry it forward, or stumble when separate decisions and tools have to work together. Reflection says Beam’s setup included persistent memory, more than 70 tools and delegated tasks, so those hand-offs were part of the test.

This was a company-run evaluation in its own offices, not proof that every agent will behave the same way. But the examples show why testing a single prompt or attack is a thin measure of readiness for agents given ongoing jobs and access to systems.

Our read

The interesting result is not simply that Beam passed some attacks. It is that the same agent could hold a rule under pressure and still absorb a convincing falsehood. For anyone building or buying agents, test what happens after the first decision too: what the system remembers, what it checks, and when a person is brought back into the loop. The machine may be running a vending business, but the reliability problem is not confined to snacks.

What to watch

  • Whether Reflection publishes results from the later, multi-participant evaluation phase it says has not yet started.
  • How the company measures false information carried into memory and the need for human intervention.
  • Whether similar testing is reported for agents operating beyond controlled workplace settings.

Discussion spark: For an agent trusted with ongoing tasks, which failure should matter more: breaking a clear rule, or believing and acting on convincing false information?

Sources and evidence

Independent WittyWires tracker for public updates about Reflection AI. Not affiliated with or endorsed by Reflection AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.