PACE cuts linear-attention model memory with four-bit recurrent states
Patrick Haller, Filipe Laitenberger and Alan Akbik introduced PACE, a post-hoc quantizer that stores recurrent states at four bits for linear-attention language models. The authors say PACE rounds states only every few generated tokens, shares scales within columns, and retains eight-bit precision for error-prone heads. On four hybrid models ranging from 4B to 35B parameters, the authors report that mixed four- and eight-bit PACE is competitive with eight-bit DAMP on the three larger models while using about two-thirds as much memory.
Lower-precision recurrent-state storage could allow more sequences to share an accelerator, while the paper reports competitive quality and preserved decoding speedups on several hybrid models.
Computer Science > Computation and Language
arXiv:2610.29545 (cs)
[Submitted on 21 Oct 2026]
Title:PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
Authors:Patrick Haller, Filipe Laitenberger, Alan Akbik
View a PDF of the paper titled PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States, by Patrick Haller and 2 other authors