开发者推出源码可查的RK3588 NPU推理运行时
开发者happyme531推出open_rknpu2_lm_infer，这是一款面向Rockchip RK3588 NPU的本地大语言模型推理引擎，无需Rockchip专有用户态推理库。其主机层代码使用Rust编写，NPU算子使用C++。该项目支持Qwen3.5-4B图像输入、可选MTP推测解码、OpenAI兼容API、流式输出、工具调用和前缀缓存。在开发者发布的对比中，启用MTP1时，处理512个token后的解码速度为10.61 tok/s，RKLLM 1.3.1为5.54 tok/s。报告未说明测试所用开发板、内存配置或驱动版本，并征集不同板卡和配置下的对比结果。
可读、可修改的实现为RK3588开发者提供了使用NPU的另一种方式，无需依赖Rockchip专有用户态推理库。其图像输入支持、OpenAI兼容接口和公布的基准数据使其与边缘LLM实验直接相关，但相关性能仍需在不同硬件和软件配置下测试。
If you have an RK3588 board, here’s another way to put its NPU to work.
I made open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU.
It runs without Rockchip’s proprietary userspace inference libraries. The upper part is written in Rust and the NPU operators are coded with C++. No black-box, you can actually read, modify, and hack the host inference code and NPU operator execution.
And now supports Qwen3.5-4B with image input and optional MTP speculative decoding. And it's fast!
Github: https://github.com/happyme531/open_rknpu2_lm_infer
- Quick comparison against RKLLM 1.3.1
#open_rknpu2_lm_infer#Rockchip#RK3588#Qwen3.5-4B#RKLLM 1.3.1#Qwen3-4B#Granite 4.2 3B#MiniCPM5-2B#MiniCPM-V 4.7 35a3