UK AI Safety Institute says safety checks must start before release

Frontier AI systems may pose safety risks while they are still being trained, tested or used internally, the UK AI Security Institute says. Its warning is a call to move safety work earlier, as current evaluation methods struggle to keep pace with increasingly capable models.

UK AI Security Institute Watch analysis

What happened

The institute says risks can emerge before a model reaches the public, particularly where autonomous systems can carry out cyber tasks. It also says established benchmarks are becoming less useful as leading models score close to the top, while new evaluations take time to build for complex tasks, multi-agent settings and longer runs.

In a blog post covered by BankInfoSecurity, the institute set out priorities including assessing and documenting risks throughout a system’s lifecycle, maintaining human oversight, defining unsafe agent behaviour, sharing information and enabling independent evaluation. It also called for better measurement of capabilities and alignment in security-relevant areas.

Why it matters

A release-day safety check cannot answer every question about a model that has already been used during training or evaluation. The institute’s point is that oversight needs to follow the system through those earlier stages too, when access may be narrower but the consequences of weak controls can still be real.

Its proposed measures are best-practice priorities, not binding rules. That leaves an awkward gap between what the institute says companies should do and what they must do.

Our read

The useful shift here is from “is this ready to launch?” to “are we evaluating and controlling it at every stage?” That is a more realistic question for autonomous systems, and a less comfortable one for companies used to treating launch as the main safety gate. The hard part will be showing that these checks work, rather than merely documenting that they happened.

What to watch

• Whether AI companies adopt lifecycle risk assessments and oversight in practice.
• How independent evaluators gain access to models and relevant testing information.
• Whether new security evaluations can keep up with more complex and longer-running systems.

Discussion spark: Should AI companies be required to follow safety checks throughout training and internal testing, or should those measures remain voluntary best practice?

not affiliated with or endorsed by UK AI Security Institute