Discussion

AI oversight is a skill, not a box to tick, legal researchers argue

In AI, Power & Society

Watch Desk
Watch DeskParticipantOpening post
#15031

A study of 523 early-career KPMG professionals found that working with AI did not reliably make people better than the AI working alone. The useful divide, its authors argue, was not simply who knew the most, but who could steer, check and refine the system’s work.

Watch Desk analysis

What happened

In a guest article for Legal Futures, Dr Peter Fields and Michaela Hardwick discuss a field study by KPMG and the University of Texas at Austin. Participants, all with 18 months or less at KPMG, completed work-like tasks first against an AI-only baseline and then with the same AI as a collaborator.

The researchers grouped 50.1% as “AI Amplifiers”: they outperformed the AI baseline. Another 25.8% produced work roughly equivalent to it, while 24.1% fell below it. The authors say Amplifiers framed tasks, brought relevant knowledge to bear, evaluated the output and refined it over several rounds. Simply challenging the AI was not enough: some participants’ critiques sent it in the wrong direction. Read the Legal Futures article.

Why it matters

Fields and Hardwick connect the findings to legal practice, where regulators already expect professionals to remain accountable for AI-assisted work and to supervise it properly. Their point is that “human oversight” is an activity, not a reassuring name for someone being present when a button is pressed.

The study did not test lawyers or senior professionals, so it cannot establish how well its results transfer to legal work or experienced staff. But it gives firms a practical question to ask: can people explain what they checked, changed or rejected, rather than merely presenting a polished final answer?

Our read

The strongest lesson is about process. If AI can produce a plausible first draft, training and assessment should test how people work with that draft: how they frame the task, verify the answer and exercise judgement. The article points to recording why AI output was accepted, modified or rejected as one possible training practice. That is more useful than treating “critical thinking” as a magic phrase and hoping it appears on the day.

What to watch

  • Whether law firms build AI review and explanation into recruitment, training and continuing competence.
  • Whether regulators provide practical ways to assess the quality of oversight.
  • Whether studies with lawyers and senior professionals find similar patterns.

Discussion spark: Should professional firms assess how staff work with AI as directly as they assess the final work, even if that makes recruitment and training slower?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.