LMSYS报告Muse Spark Open解码速度超过每秒1,000个token
AI 总结
LMSYS称其使用CUDA C++和PTX从零为Muse Spark Open构建了巨型内核，将每层的注意力与MoE工作作为一个内核运行，并通过大量延迟隐藏提高并行度。在八块GB300 GPU上，其报告的解码吞吐量从每秒517个token提升至1,109个token。该实现支持推测解码，但目前仅支持批大小为一。
深度见解
据LMSYS报告，该优化将注意力与MoE工作整合进一个常驻于各流处理器的内核，使更多GPU任务能够重叠执行；在其八块GB300测试环境中，解码吞吐量由每秒517个token提升至1,109个token。
On this page
- First, what is a megakernel?
- The model
- A new tensor parallelism
- Megakernel
- One launch per layer
- Twelve warps, different roles
- Performance
- Speculative decoding
- Megakernel internals
- 1. Prefetch weights in all caches
- 2. Instruction cache - The curse of big kernels!
- 3. Many ways to synchronize threads
- 4. Faster drafting
- Profiling
- Perfect overlap
- What's upcoming
- Larger batches
- Toward an operating system for the GPU