Developer introduces source-available RK3588 NPU inference runtime
Developer happyme531 introduced open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU that runs without Rockchip's proprietary userspace inference libraries. Its host code is written in Rust and its NPU operators in C++. The project supports Qwen3.5-4B image input, optional MTP speculative decoding, an OpenAI-compatible API, streaming, tool calling and prefix caching. In a developer-published comparison, the engine with MTP1 recorded 10.61 tokens per second for decoding after 512 tokens, compared with 5.54 tokens per second for RKLLM 1.3.1. The report does not specify the test board, RAM configuration or driver versions and invites comparisons across boards and configurations.
The readable and modifiable implementation gives RK3588 developers another way to use the NPU without Rockchip's proprietary userspace inference libraries. Its image-input support, OpenAI-compatible interface and published benchmark table make it directly relevant to edge-LLM experimentation, although the reported results still require testing across hardware and software configurations.
If you have an RK3588 board, here’s another way to put its NPU to work.
I made open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU.
It runs without Rockchip’s proprietary userspace inference libraries. The upper part is written in Rust and the NPU operators are coded with C++. No black-box, you can actually read, modify, and hack the host inference code and NPU operator execution.
And now supports Qwen3.5-4B with image input and optional MTP speculative decoding. And it's fast!
Github: https://github.com/happyme531/open_rknpu2_lm_infer
- Quick comparison against RKLLM 1.3.1
#open_rknpu2_lm_infer#Rockchip#RK3588#Qwen3.5-4B#RKLLM 1.3.1#Qwen3-4B#Granite 4.2 3B#MiniCPM5-2B#MiniCPM-V 4.7 35a3