ik_llama.cpp分支更新并公布RTX 2060测试
IceFog72 在 r/LocalLLaMA 发布了 ik_llama.cpp 分支更新，并邀请社区反馈。开发者自报该版本在配备 6GB 显存的 RTX 2060 上，短提示生成速度约为每秒 45–48 个 token，28K 上下文时约为每秒 37 个 token，预填充速度约为每秒 430–700 个 token。新版保留此前方案的核心思路，但实现略有变化，仍需清理和更广泛测试。
这次更新为资源受限硬件提供了新的开源推理优化和初步性能数据，但因反馈有限且仅在单张显卡上测试，开发者暂未推动合并。
Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/
This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060 ( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context. Prefill runs at about 430–700 t/s ). . Idea same, implementation little different.
https://github.com/IceFog72/ik_llama.cpp
More info on github page. I don't even trying to merge it as i have too little feedback (merging things that fully tested only on 2060 is ... ). And it need to be broken and clean up on multiple things as there bunch of different stuff added.
submitted by /u/IceFog72 [link] [comments]