Discussion

A bigger pile of prompt-injection attacks made one AI agent easier to hijack

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#15114

Collecting more successful prompt-injection attacks did not necessarily make an AI agent safer: in three of four tests, retraining on the larger haul left the model more vulnerable. Researchers tested a different approach that aimed for variety in attack techniques, not just a bigger pile of examples.

Watch Desk analysis

What happened

The study compared two automated attack tools against Qwen3-4B across 560 practice tasks. One tool found attacks that worked on 68.75% of tasks, compared with 33.39% for the other, and returned 7,132 working attacks rather than 4,282. Yet after retraining on those collections, the model trained on the larger set was hijacked more often in three of four test setups. In the hardest reported comparison, it failed on 19 of 40 tasks, against 11 for the model trained on the smaller collection.

The researchers’ explanation is that the larger collection contained many variations of a few techniques rather than broader coverage. They also built PIPETTE, which rewards finding successful attacks that use techniques not already represented. On the public AgentDojo test bed, ordinary injection attacks succeeded against PIPETTE-trained agents on less than half a per cent of attempts on average. Adaptive attackers making around 250 attempts reached up to 10% on the same 40 weak spots. Help Net Security reports the research was conducted by teams at the University of Waterloo, Zhejiang University and KTH Royal Institute of Technology.

Why it matters

A red-team tally can sound reassuring while counting the same basic trick in several outfits. For teams testing AI agents that read emails, webpages or tool outputs, the findings suggest that the range of attack techniques matters, not only the number of successful attacks collected.

Defence also has to leave the agent able to do its proper job. In one comparison, a PromptGuard-protected agent completed the user’s task only 16.19% of the time, while the PIPETTE-trained agent completed it 58.10% of the time. Blocking suspicious input is less useful if the useful work gets blocked with it.

Our read

This is a practical challenge to a tempting security metric: more attacks found does not automatically mean more ground covered. The results make technique diversity and task completion worth asking about when evaluating agent defences. They are not a universal verdict: the tests used a small set of open-weight models and tasks, and the authors say they have not checked their technique-coverage grader against human judgement.

What to watch

  • Whether the technique-diversity result holds across more models, tasks and independent tests.
  • Whether future evaluations measure both attack resistance and successful completion of the user’s task.
  • Whether vendors disclose the variety of attacks in their testing, rather than only the total count.

Discussion spark: When judging an AI agent’s security, should the headline metric be how many attacks it resists, or how much useful work it can still complete?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.