MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing
The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.
The analysis provides a framework for understanding why larger MoE batches can improve compute utilization while increasing all-to-all communication, padding waste and expert-load skew. It also shows why DeepEP, DeepGEMM and EPLB address different constraints that must be considered together when optimizing effective throughput under an SLO.
沿着 batch 这条主线,分析专家覆盖率、通信量、tile 填充和负载倾斜,并用端到端实测校验理论预算。
本文为「推理 INFRA 方法论」系列第 6/8 篇。发布日期按 2026 年 1–11 月编排;内容统一整理于 2026-11-01,包含年内后续更新的模型与实验资料。
Shape 与稀疏度的演变
| 模型 | E(routed) | k | 专家 inter | 稀疏度 | 形态 |
|---|---|---|---|---|---|
| 稠密 FFN (Llama-70B) | — | — | 28672 | 100% | 一个大 GEMM |
| DeepSeek-V3 (2024) | 256 | 8+1s | 2048 | 5.5% | 细粒度 + shared |
| Qwen3.5 (2026) | 512 | 10+1s | 更小 | 4.3% | 超细粒度 (shared 去了又回) |
| DS-V4-Pro (2026) | 384 | 6+1s | 2048 | 3.1% | + 原生 FP4 |
趋势:E 涨 64×、单专家缩小 14×、稀疏度 25%→3%。
后果:单专家 GEMM 从”大而稠”变”小而碎”——MoE 的一切系统问题(padding、EP、EPLB)都源于这个形态变化。
#DeepEP#DeepGEMM#EPLB#NVIDIA H20#Qwen3.5-397B-A17B-FP8#DeepSeek-V3#Llama-70B#DS-V4-Pro#SGLang#CUDA Graph