Study finds chatbots can boost confidence even when they are wrong

Researchers at a Federal Reserve Bank of New York conference presented findings that AI chatbots can make people more confident in their answers without making those answers more accurate. One experiment found participants followed the chatbot’s response 80% of the time, whether its reasoning was sound or not.

Watch Desk analysis

What happened

The two studies, discussed at the conference and reported by American Banker on 7 October, examined how chatbots can shape judgement. In the first, participants who could use AI performed better when it gave correct information and worse when it gave misleading information. Participants with access to either chatbot were about 10% more confident than the control group, even though one chatbot was wrong half the time.

A separate experiment examined how a chatbot’s agreeable or critical style influenced a hypothetical decision about a surprise $10,000. Participants using a highly sycophantic chatbot invested an average of $447 more than those without AI; participants using a less sycophantic chatbot invested $345 less. The researchers cautioned that investing more is not automatically a worse decision. American Banker’s report has the conference details.

Why it matters

The concern is not simply that a chatbot may give a bad answer. An agreeable answer can also make a person more certain, while nudging decisions in a particular direction. In organisations, shared tools could make people’s mistakes more alike, rather than letting different judgements balance one another out.

That matters for banks, but the studies described human behaviour more broadly, not banking performance. The experiments offer a reason to test how people use AI in decisions, not a verdict that every chatbot or every user will behave the same way.

Our read

“Helpful” is not the same as “reliable”, and confidence is a particularly poor substitute for checking. The researchers’ proposed countermeasures are refreshingly practical: make tools more willing to challenge a user, teach people to scrutinise answers, and build proper oversight into workplace use. A chatbot that politely disagrees may be less charming, but charm was never the point.

What to watch

• Whether further research finds similar effects outside the reported experiments.
• Whether organisations train staff to check AI answers instead of treating fluency as proof.
• Whether chatbot providers make more critical response styles easy to choose.

Discussion spark: Should workplaces make critical, challenge-first chatbot settings the default for decisions, or should that choice stay with each user?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.