Study finds language models handle word order differently from the human brain

A new study finds that language models’ internal representations can miss who did what to whom when the words stay almost the same. In an fMRI experiment, the human brain tracked those role swaps more like models built to represent grammatical structure than like standard transformer embeddings.

Watch Desk analysis

What happened

The researchers tested 108 sentences, including pairs that used nearly identical words but swapped the grammatical roles. Thirty participants took part in the brain-imaging experiment. PsyPost reports that on those same-vocabulary pairs, transformer representations showed a negative relationship with brain activity, while two models that explicitly represented sentence structure showed modest positive alignment.

Across all sentence pairs, where many used different vocabulary, the results were less stark: all four model types aligned positively with language-network activity, and the difference between the word-averaging and transformer models was not statistically significant. In a separate online study, 486 people rated sentence similarity. Transformers aligned well with human ratings overall, but the hybrid model did best on the same-vocabulary pairs.

Why it matters

A chatbot can use grammar to answer a question without its internal representations organising meaning in the same way as a person’s brain. That distinction matters when claims about machine “understanding” lean on fluent answers or broad benchmark scores. Here, changing who performed an action made a measurable difference to people, while transformer embeddings treated some role-swapped sentences as much like sentences with an added time or place detail.

The result is not a verdict on whether language models understand language, nor a test of how useful they are in everyday tasks. The reported correlations were small, the fMRI sample was modest, and the authors say practical implications were beyond the study’s scope. Those limits are part of the finding, not a footnote to wave away.

Our read

This is a useful probe of what language models represent, rather than a grand brain-versus-machine showdown. The most interesting point is that a model’s ability to produce a sensible answer does not guarantee that its internal map of sentence meaning resembles ours. The next step is to see whether that gap matters for real tasks, not just carefully constructed sentences in a scanner.

What to watch

• Whether other studies reproduce the role-tracking result with different sentences and participants.
• Whether structural representations improve performance on practical language tasks, not only similarity tests.
• How future work separates linguistic structure from visual, memory and task effects in brain scans.

Discussion spark: Should claims that language models understand language depend on how their internal representations work, or mainly on whether they reliably handle meaning in real tasks?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.