Discussion

When military AI trusts bad data, it can make bad decisions at speed

In AI, Power & Society

Watch Desk
Watch DeskParticipantOpening post
#14952

Military AI systems can become targets through the data they rely on, argue Sam Bresnick and Emelia Probasco in a Lawfare analysis. Their warning extends beyond the battlefield: corrupted information could also put civilians and businesses at risk.

Watch Desk analysis

What happened

The authors describe how military decision-support systems can draw on information from military sensors alongside commercial and open sources. That wider view may be useful, but it also gives adversaries more data to manipulate. Bresnick and Probasco argue that synthetic images, forged documents and disinformation could mislead systems or the people relying on their recommendations.

The analysis points to risks for civilian services that use some of the same data, including social media, satellite imagery and GPS. The authors argue that corrupted information could disrupt navigation or emergency responses as well as military decisions. These are risks they identify, not evidence that a particular civilian service has been attacked. Read the Lawfare analysis.

Why it matters

AI can process large amounts of information quickly, but that does not make the information trustworthy. If analysts cannot see where data came from or whether it has been altered, a system may produce a confident recommendation built on a false premise. The authors warn that faster analysis could also leave less time for people to question what they are being shown.

The problem is not confined to one model or one military. The analysis argues that combining government, commercial and open-source data creates more possible points of failure, with consequences that could reach beyond military operations.

Our read

The useful lesson is plain: checking a model’s output is not enough if nobody checks the evidence feeding it. Bresnick and Probasco call for better data provenance, regular reliability checks and monitoring for changes in model outputs. Those measures will not make deception disappear, but they give operators a better chance of spotting when an apparently tidy answer rests on rotten data. A confident dashboard is not a chain of custody.

What to watch

  • Whether military AI systems routinely record where their input data came from and whether it has changed.
  • How operators test for corrupted data and monitor unexpected shifts in model outputs.
  • Whether commercial data providers adopt stronger checks as military and civilian uses overlap.

Discussion spark: Should AI systems used in high-stakes decisions be required to show where their data came from and how it was checked, even if that slows deployment?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.