Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

EPLB

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing

    The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.

    掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my EPLB experienceAsk about EPLB

Nobody has shared their experience with EPLB yet.