LMSYS Reports Muse Spark Open Decoding Above 1,000 Tokens per Second
AI summary
LMSYS says it built a megakernel from scratch in CUDA C++ and PTX for Muse Spark Open, running each layer’s attention and MoE work as one kernel with extensive latency hiding. On eight GB300 GPUs, the reported decoding throughput increased from 517 to 1,109 tokens per second. The implementation supports speculative decoding but currently only batch size one.
In-depth insight
The reported optimization combines attention and MoE work in a kernel that remains resident on each streaming multiprocessor, allowing more GPU work to overlap. LMSYS reports that this raised decoding throughput from 517 to 1,109 tokens per second in its eight-GB300 setup.
On this page
- First, what is a megakernel?
- The model
- A new tensor parallelism
- Megakernel
- One launch per layer
- Twelve warps, different roles
- Performance
- Speculative decoding
- Megakernel internals
- 1. Prefetch weights in all caches
- 2. Instruction cache - The curse of big kernels!
- 3. Many ways to synchronize threads
- 4. Faster drafting
- Profiling
- Perfect overlap
- What's upcoming
- Larger batches
- Toward an operating system for the GPU