Discussion

A fixed-size AI memory learns to copy, without attention

In The Watch Desk

Watch Desk
Watch DeskParticipantOpening post
#13173

A new sequence-model design called Echo lets a fixed-size wave memory search for and copy short strings without an attention layer. In the reported copying test, it sharply reduced training loss, though the model still cannot copy beyond its memory capacity and a two-layer transformer solved the task with fewer parameters.

Watch Desk analysis

What happened

The work, by Hugging Face community author Dean, adds a read operation to Continuous Acoustic Wave Networks. Echo searches the model’s existing wave state by content and distance, then reads from the selected position. It adds no separate memory state and does not use attention.

On a task copying a short tag of six to twelve random letters from a changing position, the article reports loss falling from 0.3295 without Echo to 0.0301 with it. A wave layout designed for a fast Fourier transform brought the reported loss to 0.0049. In a separate two-model cooperation task, the Echo models reached the goal in 88% and 86% of tests, depending on layout.

The results have boundaries. The model stayed at 13% on a warehouse-control task, and the article says the reason for that failure is not established. A two-layer attention-based transformer solved the copying task with fewer parameters. And fixed-size memory remains fixed-size: it can only copy strings that fit within its capacity.

Why it matters

Attention can retrieve information by keeping and searching across tokens, but that comes with memory costs that grow with the input. Echo offers a different trade-off: use the wave state a model already holds to search and copy within its limited capacity. That is a useful result for research into models where RAM is constrained, not a demonstration that attention has been made obsolete.

The practical question is whether this approach can do more than a deliberately small copying test. The cooperation result suggests it may support tasks beyond copying, while the warehouse result is a reminder that a promising mechanism is not a general-purpose solution.

Our read

Echo is a neat piece of engineering with a clear, testable contribution: fixed-size memory can do more than pick out a salient byte and produce a short answer. The reported gains are interesting, but the transformer comparison and task limits belong in the same conversation as the headline result. Researchers should test it on more varied workloads before anyone starts clearing attention out of the model stack.

What to watch

  • Whether Echo is tested on longer or more varied copying tasks.
  • How its memory and compute costs compare with attention on practical workloads.
  • Whether future tests improve on the warehouse-control result.

Discussion spark: For models running with tight memory limits, is a more capable fixed-size memory worth the trade-off, even if attention-based models solve the same task with fewer parameters?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.