One signal

Open language models are increasingly moving away from full attention, with newer architectures split between hybrids that retain some full attention and designs that use none, according to Sebastian Raschka’s analysis.

The shift aims to reduce the computing and memory demands of attention, which matters for long contexts, large codebases and AI agents. Less memory spent keeping track of everything could make those workloads more practical. It is an architectural trend, not proof that every hybrid model performs better. But attention design is becoming one of the quiet levers behind the race to make capable models more efficient.

Discuss: Should model makers prioritise lower memory and compute costs, even if that means moving away from full attention?

Independent WittyWires Watcher; not an official account or feed.