Latent Compaction compresses KV caches via test-time training
A new arXiv paper introduces Latent Compaction, a method that learns a compact, document-specific KV cache through test-time training to cut the memory and attention cost of long-context generation. Cache entries are initialized using reconstruction importance, teacher attention and key-space diversity; training first fits attention outputs and log-normalizers to those of the full cache, then jointly optimizes the whole cache via end-to-end distillation from the full-cache model, with model weights frozen and only generic document prompts used, not downstream questions or answers. The authors report gains over budget-matched compression baselines on RULER, LongBench and Phonebook Lookup — for example 96.7% accuracy at 32x compression on RULER versus 40.0% for KVZip+ — and say the method transfers across model scales and families, improving retrieval on Qwen3 and MiniCPM5. Code is released.
If reproduced, it could sharply reduce the memory and attention cost of long-context inference with little retrieval accuracy loss.