Trade-offs in Distributed Inference for Attention and MoE
The report analyzes distributed-inference strategies across tensor, data, expert, context, and pipeline parallelism using weight, KV-cache, activation, communication, capacity, and latency budgets. Its author argues that GQA can shard KV heads up to the KV-head count, while MLA's shared latent cache must be replicated under tensor parallelism, and compares MoE tensor parallelism with expert parallelism in terms of communication, memory use, load balancing, and implementation complexity. The report title labels it installment 2, while the body calls it the third of eight installments in an inference-infrastructure methodology series. Its comparative figures and conclusions are the author's analysis and are not independently corroborated by the supplied evidence.
The report provides an engineering framework for determining how model state is partitioned, then assessing per-device capacity, communication, memory, and latency constraints when selecting a parallelism strategy for large-model inference.
围绕权重、KV、激活和通信预算，推演 Attention 与 MoE 的并行组合，并分析长上下文和流水线的额外约束。
本文为「推理 INFRA 方法论」系列第 3/8 篇。发布日期按 2026 年 1–11 月编排；内容统一整理于 2026-11-01，包含年内后续更新的模型与实验资料。
Attention/MoE组合方式
Attention-TP/DP和MLA的关系:
GQA:KV 沿 n_kv 个头切 → TP ≤ n_kv(=8 时 TP8 正好),TP 同时摊薄 KV 显存 ✅
MLA:KV 是全头共享的一个 latent → 无 head 维可切 → TP 下只能【复制】,但是tp能切weight降低计算量
TP8 = 同一份 KV 存 8 遍 → 显存爆炸 ❌
⟹ MLA 模型 attention 必走 DP:每卡管自己的请求和 KV,不复制
Attention-DP/TP和MoE-TP/EP组合下通信机制区别：