Lightweight CPU Replication Explores Kimi K3 KDA Parallelism
The second part of the author's lightweight, CPU-only replication of Kimi Delta Attention in Kimi K3 examines ShortConv, recurrent state updates, chunkwise parallelism, context parallelism and decoding caches. The author reports a KCP prefix-scan error of 1.110e-16 and 37 passing tests in their own test suite. The report also notes that larger chunks can increase memory use, while floating-point errors and numerical-stability limits constrain chunk and tile sizes.
The report provides implementation details, numerical experiments and test results illustrating the tradeoffs between within-chunk parallelism, recurrent state dependencies across chunks, and numerical stability.
【轻量化复刻大模型 / KIMI K3】KDA（Kimi Delta Attention）：并行
Some source content is omitted. Continue reading on the original site.
#Kimi K3#Kimi Delta Attention#KDACache#KDA Context Parallelism#DPLR