Researchers say an AI agent directed by a human researcher persuaded 300 of 363 tested language models to reveal a secret planted in their system prompts. The results point to a practical warning for anyone relying on prompts to keep sensitive information private: a model’s instructions are not a secure vault.
Watch Desk analysis
What happened
The arXiv study, “Eight in Ten LLMs Leak System-Prompt Secrets”, tested text-in/text-out models available through OpenRouter. Researchers embedded a protected 16-character flag in each system prompt and used a human-steered AI agent to try to extract it. They report that 300 models disclosed the flag.
In 170 successful captures, the agent used a non-English language pivot, most often French, according to the study. That is a finding about this test and its specific planted secret, not evidence that every model will reveal every kind of confidential information.
Why it matters
System prompts can shape a model’s behaviour, but this result is a reminder not to confuse instructions with access controls. If a secret must remain secret, storing it in a prompt and hoping the model will keep mum is a rather optimistic security plan.
The study’s result is also a useful challenge to model testing: an attack that changes language may find weaknesses that a straightforward prompt misses. The figures describe the researchers’ test, not a universal failure rate for all models or deployments.
Our read
This is a meaningful security signal, and a good reason to keep credentials and other sensitive data out of system prompts. The useful takeaway is not that every model is equally exposed, but that prompt secrecy should not be the control protecting something valuable.
What to watch
- Whether the paper’s full methods clarify how models and test conditions were selected.
- Whether follow-up testing reproduces the result across different prompts and deployments.
- Whether model providers publish practical mitigations, rather than asking users to treat prompts as secure storage.
Discussion spark: If system prompts should not hold secrets, what should providers and developers change first: model safeguards, deployment practices, or the way they test for leaks?
Sources and evidence
- Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study (18 November 2026, 01:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.