Scale’s new chemistry benchmark finds AI models struggle with dual-use risk

Scale AI has introduced DualChem, a 600-question benchmark for testing how AI models handle chemistry questions that could have beneficial or harmful uses. Scale says its tests found models often kept answering even when they flagged a question as potentially dual-use, a gap with real implications for assessing AI risk.

Watch Desk analysis

What happened

DualChem was developed by Scale with the Center for Advancing Innovation and Standards for Super Intelligence, part of the US National Institute of Standards and Technology. Its questions cover areas including molecular design, synthesis procedures and information management, and were curated by experts and reviewed by PhD-level evaluators.

Scale says it assessed roughly 30 frontier models. The company reports that they were generally about 50% accurate on dual-use chemistry questions, with the top performers reaching roughly 67%. It also found significant variation between models’ assessments of question severity and expert judgements. In many cases, models continued to answer even after identifying a question as potentially dual-use. Read FedScoop’s report on DualChem.

Key findings

• A substantial test set
  The benchmark contains 600 expert-curated chemistry questions reviewed by PhD-level evaluators.
• Mixed performance
  Scale says models were generally around 50% accurate, with top performers at roughly 67%.
• Risk flags do not reliably change answers
  Models often continued responding after recognising a question as potentially dual-use.

Why it matters

Chemistry questions can sit in an uncomfortable middle ground: the same information may support useful research or harmful activity. Tests focused only on explicit requests for dangerous instructions may miss how models handle ambiguous, realistic questions. DualChem is intended to probe that harder territory, and to give government evaluators a tool of their own.

The reported gap between recognising risk and changing a response matters because a warning label is not the same as a safeguard. If a model identifies a question as risky but answers anyway, the key question is whether its risk judgement affects what it tells the user.

Our read

DualChem puts a useful evaluation problem in plain view: models need to do more than spot a risky question; their responses should reflect the risk. The reported results are Scale’s assessment, not an independent verdict on every model. Still, a purpose-built benchmark for ambiguous chemistry queries gives researchers and public agencies something more concrete to scrutinise than broad assurances about safety.

What to watch

• Whether Scale or CAISSI publishes further detail on the benchmark and its evaluation methods.
• How different models perform when tested against the same questions over time.
• Whether the benchmark’s results lead to changes in model responses or government evaluation practice.

Discussion spark: If an AI model flags a chemistry question as potentially risky but answers it anyway, should that count as a safety failure even when the answer is not demonstrably harmful?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.