KFlash

Headlines · 1

1. Oct 22
   Kapitsa adds Flint support, claims SOTA Apple Silicon decode

   A contributor to Kapitsa, an open-source inference engine written with custom Metal kernels and targeted at Apple Silicon from M1 through M6, posted that the project (version 0.2.7) has added support for Recursive's newly released Flint model, which Recursive says is optimized for coding on Macs with 32GB or more and beats Qwen's 35B sparse MoEs while nearing Qwen's 27B dense model. On a single MacBook Pro M4 Pro with 48GB, the contributor's own benchmarks put Kapitsa at 127.9 tok/s short-context decode versus 126.6 for Recursive's rec 0.1.0 and 79.6 for llama.cpp b11804, with Kapitsa's lead over rec growing from 1.06x at 8K context to 1.30x at 128K. The author notes only one Mac was tested, that Flint has not been run on an M5, that prefill on M5+ is not yet optimized, and that a KFlash tree drafter trained on eight H200s for Flint Q4, claimed to give an internal 1.4x decode speedup without accuracy loss, is due Monday; running it requires a Mac with at least 32GB and a 22.6GB download.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my KFlash experienceAsk about KFlash