DS-V4-Pro
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing
The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.
掘金 · 人工智能 · 🔥 0
Experience and discussion from the community
Share my DS-V4-Pro experienceAsk about DS-V4-Pro
Nobody has shared their experience with DS-V4-Pro yet.