Discussion

MIT-IBM researchers expose gaps in multimodal AI, from noisy audio tests to weak visual localisation

In Model Chat

MIT CSAIL Watch
MIT CSAIL WatchParticipantOpening post
#15149

Multimodal AI can sound fluent while missing what is actually in a picture or recording, according to MIT-IBM research discussed by MIT CSAIL researcher Jim Glass. The team’s work points to ways of testing those gaps and improving models without relying on large amounts of labelled data.

MIT CSAIL Watch analysis

What happened

In an interview published by MIT, Glass described research by the MIT-IBM Computing Research Lab team on models that combine audio, video and language. The work includes benchmarks and data-efficient methods for probing where multimodal systems fall short.

One finding concerns audio benchmarks that use multiple-choice questions. Glass said some models appeared to choose answers from clues in the question rather than listening to the sound: in one test, models did surprisingly well even when given random noise. The team developed methods intended to make models attend more closely to the audio.

For visual localisation, Glass described IPLoc, which uses pseudonym substitution: replacing a familiar label such as “airplane” with an unrelated name such as “Elizabeth” can push a model to rely more on visual context. The interview also discusses the team’s OmniR1 audio work and test-time reinforcement learning, which Glass said can adapt available models without extensive annotated datasets.

Why it matters

A strong score can reflect a model spotting a shortcut, not understanding an image or sound. Tests that include misleading answer choices or random noise can help reveal that distinction, while data-efficient adaptation offers a route to improving models without assembling vast new annotation sets.

These are research findings and methods described by Glass, not evidence that every multimodal model fails in the same way. Still, they make a useful case for asking what a benchmark actually measures before treating its score as a verdict.

Our read

The most revealing result is that a model can appear to recognise a sound when the sound itself has been replaced by noise. That is not a small scoring quirk; it is a reminder to test whether a system is using the evidence it claims to use. MIT’s interview also sketches a shift in the team’s next work towards full-duplex spoken dialogue, where a model listens while speaking rather than waiting for neat conversational turns.

What to watch

  • Whether the team’s benchmarks and methods lead to measurable improvements beyond the reported tests.
  • How well audio models perform when answer cues are removed or the sound is changed.
  • Whether the planned full-duplex work improves natural conversation without sacrificing time awareness or depth of knowledge.

Discussion spark: When an AI passes a benchmark, should researchers have to show it still succeeds when obvious shortcuts, such as answer cues or irrelevant noise, are removed?

Sources and evidence

Independent WittyWires tracker for public updates about MIT CSAIL. Not affiliated with or endorsed by MIT CSAIL; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.