Kimi Delta Attention

Headlines · 3

1. 2d
   CPU-Only Reproduction of Kimi K3 Gated MLA

   A technical write-up presents a lightweight, scaled-down reproduction of Kimi K3's Gated Multi-head Latent Attention that runs on CPU without GPUs or large-scale training. The project covers latent compression and reconstruction, NoPE alongside interleaved KDA layers, a full-rank output gate, MLACache, FP32 accumulation, and incremental-decoding tests, with Gradio demos that can run without a GPU. The author says the implementation aligns with selected Kimi K3 code paths, but the supplied report does not include independent performance or equivalence measurements.

   掘金 · 人工智能 · 🔥 2
2. 7d
   Interactive Demo Explores Kimi K3's KDA Attention Module

   RossiLH published a runnable Gradio demo that presents a lightweight reproduction of the Kimi Delta Attention module in Kimi K3. It covers ShortConv, SiTU-GLU, recurrent updates, bounded decay, chunked parallelism, KCP prefix scanning, a complete KimiDeltaAttention layer, and the stated 3:1 KDA-to-MLA scheduling through 90 test functions across 15 tabs. The project runs on CPU and uses reduced or approximated components when no open-source implementation is available.

   掘金 · 人工智能 · 🔥 0
3. Nov 3
   Lightweight CPU Replication Explores Kimi K3 KDA Parallelism

   The second part of the author's lightweight, CPU-only replication of Kimi Delta Attention in Kimi K3 examines ShortConv, recurrent state updates, chunkwise parallelism, context parallelism and decoding caches. The author reports a KCP prefix-scan error of 1.110e-16 and 37 passing tests in their own test suite. The report also notes that larger chunks can increase memory use, while floating-point errors and numerical-stability limits constrain chunk and tile sizes.

   掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my Kimi Delta Attention experienceAsk about Kimi Delta Attention