Watch Desk posted an update
AI tools may deepen discrimination against minorities when they learn from biased records and operate without oversight, researchers warn, according to the BBC.
Why it mattersIt is a concise but important reminder: automated decisions can carry old biases into new settings, rather than leaving them behind.
Discuss: Should oversight of AI systems focus on auditing the records they learn from, or on checking the decisions they make in practice?
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk
Watch Desk Update What changedThe BBC’s new account adds detail on how predictive policing can turn past patterns into future records. When police patrol areas flagged by a system more heavily, more crime may be recorded there; that data can then make those areas look like still stronger candidates for patrols. The result is a feedback loop, not simply a forecast that sits on a screen.
Sociologist Sarah Brayne’s research inside the Los Angeles Police Department found that data used to predict crime was skewed: crimes in public places were more likely to be recorded, and police attention fell disproportionately on Black neighbourhoods.
The piece includes a counterpoint from AI researcher Dimitrios Kollias: modern systems can do more than reproduce training examples, and bias is not inevitable. He says systems can be tested for unfair outcomes and safeguards introduced.
Sources and evidence
- AI in policing risks deepening discrimination, research warns - BBC: The BBC reports that biased or unlawful police records may feed predictive systems and reinforce existing patterns of surveillance; it also reports researchers’ findings on skewed LAPD data and police datasets from nine of 13 US areas.
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk Update What changedThe BBC’s new account adds examples of how biased records can shape AI decisions beyond predictive policing. Researchers at New York University found that, in nine of the 13 US areas they examined, police data from periods of unlawful or biased policing was available to train systems.
The essay also points to a 2019 healthcare study in which an AI system used expected care costs to identify patients needing extra support. Because less was already being spent on Black patients who were equally ill, the system favoured white patients.
The new examples sharpen the practical question: oversight is not only about checking a model’s output, but also where its records came from and what it is being asked to predict. The BBC includes a counterpoint from AI researcher Dimitrios Kollias, who says bias is not inevitable and systems can be tested and safeguarded.
Sources and evidence
- Police using AI risks making discrimination worse, researchers warn - BBC: The BBC essay adds reported examples of biased police records used in AI systems and a healthcare algorithm that selected fewer Black patients for support when it predicted costs rather than illness.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.