LMSYS says it built a megakernel from scratch in CUDA C++ and PTX for Muse Spark Open, running each layer’s attention and MoE work as one kernel with extensive latency hiding. On eight GB300 GPUs, the reported decoding throughput increased from 517 to 1,109 tokens per second. The implementation supports speculative decoding but currently only batch size one.
LatentLoop compresses the sequence during a looped Mixture-of-Experts model's first pass, keeps the second at full resolution, and uses the saved compute to increase the hidden size. It shares MoE weights across passes while using separate attention parameters, and the authors report lower validation NLL across 15 configurations from 1.3B to 7B parameters, along with improved average scores on nine benchmarks.
What problem it solves
Selecting an open-source language model requires accounting for available memory, processor capabilities, GPU configuration, model size, quantization, and context requirements. The project addresses the difficulty of determining which models can run comfortably on a particular machine and comparing fit, speed, quality, and context before deployment.
Key features
Hardware aut…
Optimizing a GPU kernel or completing a set of formal proofs often requires repeated trial and error: try a different implementation, run an evaluation, and keep iterating in whichever direction proves effective. When giving a model more test-time compute, the most direct approach is to let more agents try simultaneously; if the task only needs one best result, the value of the other attempts lies…
What problem it solves
Running frontier-scale open-weight models locally requires an inference platform that can make efficient use of the heterogeneous hardware available on a personal device. The project addresses this by treating GPUs, CPUs, host memory, and interconnects as a unified, elastic inference platform.
Key features
Edge-native MoE runtime — The runtime provides MoE serving with bandw…
What problem it solves
LLM and Visual Gen inference requires specialized kernels, efficient runtime orchestration, and flexible deployment across NVIDIA GPU configurations. Developers also need a modular framework that supports customization while integrating with established inference tooling.
Key features
Optimized inference kernels — Custom kernels accelerate common inference operations, includ…
llmfit• Pinned
