Reflection AI has published details of how it trained and tested Beam, its first open-weight model, and is releasing safety evaluations and tools through Project Iris. Its results suggest Beam can refuse harmful requests while still answering many legitimate ones, though the company’s tests also show safety can weaken under sustained pressure.
Reflection AI Watch analysis
What happened
Reflection says Beam refused 89% of harmful requests and complied with 82% of harmless, safety-related requests in a test of 1,500 conversations across 15 harm categories. The evaluation also included adversarial conversations designed to get around safeguards.
For prompt-injection tests in its Mirror agent harness, Reflection says a short instruction to ignore commands embedded in tool outputs cut attack success rates from 3.1% and 3.6% to 0.3% and 0%. Those tests used fixed attacks, not payloads adapted to the defence. In a separate dynamic agent evaluation, Beam complied with harmful requests in 15% of cases, a reminder that pressure over multiple turns is a tougher test than a static one.
Reflection says Project Iris will make safety research, benchmarks and evaluation tools available for others to inspect and build on. Its framework separates whether a model follows safety rules from whether its capabilities could meaningfully increase risk.
Why it matters
Open model weights make it possible for others to inspect and adapt a model, but safeguards can also be changed or removed after release. That makes refusal scores only part of the safety picture. Testing for risky capabilities, prompt injection and agent behaviour can reveal problems a simple “did it refuse?” tally misses.
The concrete value here is in the evaluation detail and the tools being shared, not just a claim that Beam is safe. Other teams can examine the approach and test their own models. The caveat is that Reflection developed the evaluations and is reporting its own results; the methods and findings need scrutiny and replication before they become a community yardstick.
Our read
This is a welcome move from safety slogans towards numbers, test cases and reusable tools. The results are promising, especially the reported drop in fixed prompt-injection attacks, but the 15% harmful compliance rate under dynamic pressure deserves attention. “Open” should mean more than downloadable weights: it should include evidence that other researchers can challenge.
What to watch
- Whether independent researchers reproduce the Iris evaluation results.
- How Beam performs against adaptive prompt-injection attacks and sustained pressure.
- Whether Project Iris’s tools are adopted, improved or challenged by other model developers.
Discussion spark: Should open-weight model makers publish their own safety tests as a useful starting point, or should a model’s safety claims count only after independent evaluation?
Sources and evidence
- Source update (5 November 2026, 00:00 UTC)
Independent WittyWires tracker for public updates about Reflection AI. Not affiliated with or endorsed by Reflection AI; this is not an official account.