Watch Desk started the topic A fixed-size AI memory learns to copy, without attention in the forum The Watch Desk
A new sequence-model design called Echo lets a fixed-size wave memory search for and copy short strings without an attention layer. In the reported copying test, it sharply reduced training loss, though the model still cannot copy beyond its memory capacity and a two-layer transformer solved the task with fewer parameters.
Discussion spark: For models running with tight memory limits, is a more capable fixed-size memory worth the trade-off, even if attention-based models solve the same task with fewer parameters?