Bottom line: Sparse Delta Memory significantly increases the state capacity of linear RNNs without raising computational costs, thereby improving long-context and reasoning performance.
Researchers introduce an architecture update for linear RNNs that increases memory size through sparse addressing by orders of magnitude while keeping computational effort constant. This particularly improves performance on long-context tasks.
Linear Attention models offer constant state size and consistent computational costs per token. Their disadvantage compared to softmax transformer architectures: limited state size leads to weaker long-context recall. While a larger state improves retrieval accuracy, it requires more computational operations (FLOPs).
The new Sparse Delta Memory (SDM) method resolves this dilemma through a sparse addressing mechanism: it replaces the dense key-value matrix multiplication in the Gated-DeltaNet architecture with sparse read and write accesses to a large explicit memory. This allows memory capacity to be increased by orders of magnitude without wasting compute time.
Under identical FLOP budgets and parameter counts, SDM demonstrates significant improvements on in-context learning and long-context retrieval tasks. An additional trick: when the initial memory state becomes trainable (parametric memory), the model also benefits on common knowledge and reasoning tasks.
Source: arxiv.org · Published 7 July 2026
Lumi AI News — AI-assisted curation according to Article 50 EU AI Act. Paraphrase and classification via Lumi News Pipeline v1.7.3.