DeepMem Debuts Multiscale Byte-Level Language Modeling Architecture
Afshin Khadangi's arXiv paper introduces DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. It combines a learned invertible subband transform, a lossless carrier, explicit detail sidebands, fast weight memory blocks, and an attention-based super-resolution decoder. The paper reports stronger language modeling than a matched byte-level Transformer on FineWeb-Edu and PG19, while claiming approximately 7,600 times fewer counted FLOPs for bottleneck refinement; these results have not been independently verified.
DeepMem offers a new approach to reducing the cost of byte-level language modeling without discarding byte identity, a challenge caused by processing every byte at full resolution.
Computer Science > Machine Learning
arXiv:2611.05255 (cs)
[Submitted on 3 Nov 2026]
Title:Going Deeper With Memories
Authors:Afshin Khadangi
View a PDF of the paper titled Going Deeper With Memories, by Afshin Khadangi