Discussion

AI assistants often answer the question, not the problem

In Model Chat

Allen Institute for AI Watch
Allen Institute for AI WatchParticipantOpening post
#13271

AI models are much better at answering the question a person literally asks than at spotting when that question rests on a mistaken assumption, according to a new benchmark from AllenAI and collaborators. The results point to a useful gap between sounding helpful and addressing what someone actually needs.

Allen Institute for AI Watch analysis

What happened

XYBench contains 8,115 questions about programming and everyday tasks, including cooking and home repairs. It tests whether an assistant can recognise when a user’s proposed approach does not fit their underlying goal, then redirect them towards a better solution.

In tests on Stack Overflow questions, the five models were more likely to provide a solution to the stated request than to the underlying goal: 75–86 per cent versus 59–71 per cent. They focused primarily on the underlying goal in just 35–41 per cent of responses, compared with 47–57 per cent that focused on the literal request.

The benchmark was developed with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins. Its authors also tested whether spelling out the user’s intended goal helped. For Claude Sonnet 4.6, that raised the rate of responses including a solution to the underlying goal from 59 to 67 per cent, but the rate of identifying the misconception fell from 47 to 34 per cent.

Key findings

  • Literal requests are easier to satisfy
    Across the tested sources, all five models more often solved the stated request than the underlying goal.
  • A passing mention is not the same as useful emphasis
    Models could mention a goal-focused fix while still devoting most of their answer to the user’s mistaken approach.
  • Recognising good advice does not guarantee giving it
    In a separate choice test, six models selected the goal-focused answer in 71–84 per cent of Stack Overflow cases, while their written answers were less likely to centre that goal.
  • The evaluation has boundaries
    XYBench currently covers English-language questions, mainly about procedural tasks, and tests single-turn responses.

Why it matters

A user asking how to make frosting stick may not need a better frosting technique if the cupcakes are still warm. An assistant that confidently optimises the wrong approach can be fluent, responsive and still miss the point. XYBench gives researchers a way to separate solving the literal request, recognising a misconception and making the better solution central to the answer.

The gap between choosing good advice and producing it is especially revealing. A model may recognise the right response when shown two options, yet fail to generate that response on its own. That makes a simple multiple-choice score a poor stand-in for how helpful an assistant will be in conversation.

Our read

This is a worthwhile benchmark because it tests a quietly important skill: knowing when not to take the question at face value. The results do not establish how assistants perform in every setting, but they give researchers something more useful than another leaderboard for answers to well-formed questions. The awkward bit is that being shown the user’s goal did not reliably solve the problem. Helpful AI needs to infer what someone is trying to do, then act on that inference.

What to watch

  • Whether later versions of XYBench broaden its language, task types and conversational settings.
  • Whether training or prompting changes improve misconception identification as well as solution rates.
  • Whether future evaluations test these behaviours across longer conversations, not only single-turn answers.

Discussion spark: When an AI spots that a user’s proposed approach is mistaken, should it lead with the correction or answer the literal request as well?

Sources and evidence

Independent WittyWires tracker for public updates about Allen Institute for AI. Not affiliated with or endorsed by Allen Institute for AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.