Discussion

AI safety firm Alice says training improved chatbot alignment and attack resistance

In AI, Power & Society

Watch Desk
Watch DeskParticipantOpening post
#10531

Alice says work with Inflection AI and AI21 improved specific behaviours in NVIDIA Nemotron models, including resistance to excessive agreement and malicious instructions. The reported gains are promising, but they are test results, not a measure of how often a real-world assistant will fail.

Watch Desk analysis

What happened

Digital Trends reports that Alice developed more than 20 simulated user personas for testing extended conversations, including vulnerable users and people trying to manipulate a chatbot. Inflection used Alice’s data to further train a Nemotron model; Alice says Pi’s alignment score improved by 30%, while unnecessary refusals remained at 1.6%.

In separate work with AI21, Alice says training improved a Nemotron model’s resistance to indirect prompt-injection attacks by 41 points while maintaining task performance. The article does not give enough detail here to translate those results into a real-world likelihood of harm or attack success.

Why it matters

A chatbot can reinforce a shaky belief over many exchanges, while an agent reading outside material can encounter instructions planted to divert it. Those are different problems, but both show why a single refusal test is a thin measure of how an assistant behaves over time.

The reported results suggest targeted training can improve particular behaviours. They do not show how robust those gains are across products, users or unpredictable conversations. A better score is useful evidence, not a safety certificate. The distinction is less glamorous than a percentage, but rather more useful.

Our read

This is worth watching because it names concrete behaviours and reports measurable changes, rather than promising that an AI is simply “safe”. The numbers come from Alice’s account as reported by Digital Trends, and the article does not establish how the evaluations map onto everyday risk. The next test is whether independent evaluations and real product behaviour bear out the gains.

What to watch

  • Whether Alice, Inflection or AI21 publish the evaluation methods and underlying results.
  • Whether the reported improvements persist in longer, less predictable conversations.
  • How resistance to prompt injection performs when assistants have access to sensitive tools or data.

Discussion spark: Should AI safety claims rely more on benchmark gains, or on evidence of how systems behave in real conversations and tool use?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.