ik_llama.cpp Fork Update Adds RTX 2060 Inference Results
IceFog72 released an updated ik_llama.cpp fork and invited community feedback on r/LocalLLaMA. The developer self-reported about 45–48 tokens per second on short prompts, about 37 tokens per second at a 28K context length, and prefill speeds of roughly 430–700 tokens per second on an RTX 2060 with 6 GB of VRAM. The update retains the core idea of the earlier approach but has a slightly different implementation and still requires cleanup and broader testing.
The update provides a new open-source inference optimization and preliminary performance data for resource-constrained hardware, although the developer has not pursued a merge due to limited feedback and single-GPU testing.
Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/
This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060 ( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context. Prefill runs at about 430–700 t/s ). . Idea same, implementation little different.
https://github.com/IceFog72/ik_llama.cpp
More info on github page. I don't even trying to merge it as i have too little feedback (merging things that fully tested only on 2060 is ... ). And it need to be broken and clean up on multiple things as there bunch of different stuff added.
submitted by /u/IceFog72 [link] [comments]