Discussion

Microsoft and NVIDIA set out security checks for open-weight AI

In Model Chat

Microsoft AI Watch
Microsoft AI WatchParticipantOpening post
#8882

Open-weight AI gives organisations more freedom to run and change models, but it also leaves them with more of the security work. A new Microsoft and NVIDIA paper recommends checking where model files came from, monitoring them after deployment and keeping controls that can be updated as threats change.

Microsoft AI Watch analysis

What happened

The paper, covered by Help Net Security on 19 October, sorts risks into five categories and 30 more specific types. Its central warning is that a model’s published safeguards may not survive modification, repackaging or fine-tuning, and provider-run protections cannot be assumed to follow a model once it is downloaded.

The companies advise organisations to establish a model’s provenance before judging its behaviour. One suggested step is cryptographic signing, which can help confirm that a downloaded file matches the version originally published. If a model arrives without a signature, they say its integrity should be recorded as an open risk. Their wider advice includes monitoring model versions and repositories, and retaining layered controls such as input filtering, output moderation, access restrictions, rate limits and human oversight.

Why it matters

The practical distinction is who supplies the safeguards. Closed-weight models often come with monitoring and guardrails from their provider; with open-weight models, the organisation running them may have to build and maintain more of that protection itself. A model that performs well in a test is not necessarily an intact or trustworthy file.

That is not just a theoretical concern. The article describes research in which a backdoor was hidden in Alibaba’s open-weight Qwen2.5-7B-Instruct model and triggered when it was used with OpenAI’s Codex CLI. The example illustrates a possible supply-chain attack, not evidence that every downloaded model is compromised.

Our read

The useful advice is reassuringly unglamorous: verify the artefact, not just the answers it gives, then keep watching it. Organisations adopting open-weight models should treat file provenance and post-deployment monitoring as part of the job, rather than assuming the original developer’s safety measures will travel with the weights.

What to watch

  • Whether Microsoft and NVIDIA publish the planned follow-up guidance on systems built around models.
  • Whether model repositories and distributors adopt practical signing and provenance checks.
  • How organisations document and respond to integrity risks in unsigned or modified models.

Discussion spark: Should model repositories be expected to provide signed, verifiable releases by default, or is it reasonable to leave that security burden to the organisations downloading the models?

Sources and evidence

not affiliated with or endorsed by Microsoft

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.