Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing

AI summary

The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.

In-depth insight

The analysis provides a framework for understanding why larger MoE batches can improve compute utilization while increasing all-to-all communication, padding waste and expert-load skew. It also shows why DeepEP, DeepGEMM and EPLB address different constraints that must be considered together when optimizing effective throughput under an SLO.

沿着 batch 这条主线,分析专家覆盖率、通信量、tile 填充和负载倾斜,并用端到端实测校验理论预算。

本文为「推理 INFRA 方法论」系列第 6/8 篇。发布日期按 2026 年 1–11 月编排;内容统一整理于 2026-11-01,包含年内后续更新的模型与实验资料。

Shape 与稀疏度的演变

模型E(routed)k专家 inter稀疏度形态
稠密 FFN (Llama-70B)——28672100%一个大 GEMM
DeepSeek-V3 (2024)2568+1s20485.5%细粒度 + shared
Qwen3.5 (2026)51210+1s更小4.3%超细粒度 (shared 去了又回)
DS-V4-Pro (2026)3846+1s20483.1%+ 原生 FP4

趋势:E 涨 64×、单专家缩小 14×、稀疏度 25%→3%。

后果:单专家 GEMM 从”大而稠”变”小而碎”——MoE 的一切系统问题(padding、EP、EPLB)都源于这个形态变化。

#DeepEP#DeepGEMM#EPLB#NVIDIA H20#Qwen3.5-397B-A17B-FP8#DeepSeek-V3#Llama-70B#DS-V4-Pro#SGLang#CUDA Graph

Original · 8 min readRead the original
Group chat

No comments yet. Start the conversation.