KimiDeltaAttention

Headlines · 2

1. 7d
   Interactive Demo Explores Kimi K3's KDA Attention Module

   RossiLH published a runnable Gradio demo that presents a lightweight reproduction of the Kimi Delta Attention module in Kimi K3. It covers ShortConv, SiTU-GLU, recurrent updates, bounded decay, chunked parallelism, KCP prefix scanning, a complete KimiDeltaAttention layer, and the stated 3:1 KDA-to-MLA scheduling through 90 test functions across 15 tabs. The project runs on CPU and uses reduced or approximated components when no open-source implementation is available.

   掘金 · 人工智能 · 🔥 0
2. Nov 2
   CPU Reproduction Explores Kimi K3’s Kimi Delta Attention

   A Juejin AI article presents the first part of a lightweight, CPU-only reproduction of Kimi K3’s Kimi Delta Attention, without GPU use or large-scale training. It analyzes state recurrence, delta-rule writes, bounded logarithmic decay, channel-wise forgetting, head-wise RMSNorm, output gating, and comparisons with other attention and state-space mechanisms, alongside runnable implementation code and demos.

   掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my KimiDeltaAttention experienceAsk about KimiDeltaAttention