CPU-only replication examines Kimi K3 Attention Residuals
The second installment of a CPU-only Attention Residuals replication associated with Kimi K3 examines the scoring kernel, Full and Block variants, and the AttnResState lifecycle. It also introduces a two-stage online-softmax algorithm and explains production-oriented SGLang fused-kernel concepts. Components without open-source versions are replaced by approximations, and the paper's 25-configuration architecture experiment was not reproduced.
The project provides directly runnable Gradio demos without a GPU and a detailed account of Attention Residuals implementation and state management, making these mechanisms easier to inspect.
【轻量化复刻大模型 / KIMI K3】Attention Residuals 完整梳理(2)
- 0x00 概要
- 0x01 子知识点拆解
- 0x02 子知识点深度分析
- 2.0 共享抽象与设计动机
- 2.1 打分核(scoring kernel)与伪查询
- 2.2 Full AttnRes
- 2.3 Block AttnRes
- 2.4 状态机(AttnResState —— 深度历史如何在层间流动)
- 2.5 online-softmax 两阶段算法(论文 Algorithm 1)
- 2.6 生产级 fused kernel 原理(SGLang;TMA + online softmax + 通信融合)
- 2.7 对比对象 —— mHC(Manifold-Constrained Hyper-Connections)
- 2.8 基础设施:跨阶段缓存 + 高效 Prefill(论文 §4)
- 2.9 学到的深度注意模式(论文 §5.4.2,Fig. 8)
- 2.10 统一视角:残差连接作为结构化矩阵(论文 §6.2,Table 5)
- 2.11 最优架构分析(论文 §5.4.1,Fig. 7)
- 0xFF 参考
Some source content is omitted. Continue reading on the original site.
#Attention Residuals#Kimi K3#AttnResState#SGLang#mHC#Manifold-Constrained Hyper-Connections#Gradio