Local LLM inference tools multiply as users weigh speed against quality
A Reddit user says the local LLM inference ecosystem has expanded rapidly since March 2025, from established tools such as llama.cpp, Ollama, vLLM and TabbyAPI to numerous newer projects. The author reports that llama.cpp continues to improve quickly, while newer engines may prioritize speed or lower resource use at the expense of output quality; their specific model and hardware claims are not independently verified here.
The discussion reflects a practical source-supported challenge for local-model users: choosing among increasingly specialized inference engines while balancing throughput, resource consumption and generation quality.
Hey everyone,
Not a native speaker, happy to be corrected! Can only learn from it.
Back when I started out in march 2025, you had:
- llama.cpp
- ollama (repackaged llama.cpp for ease of use)
- koboldcpp (repackaged llama.cpp for roleplay)
- vllm (for professional deployments)
- exl3 + tabbyapi (for bleeding edge)
- litert (for mobile devices)
Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Ninfer, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with.
Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality).
Personally I'm stuck with llama.cpp because all those developments are simply too fast to keep up with.
llama.cpp v0.8.0 might run Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15t/s gen on dual RTX 5060 Ti 16GB (PCIE 5.0 x8x8, tensor parallel, no MTP) with dual 48GB DDR5-6000MHz, but at least I know the quality will be decent.
But yeah, what you use? How do you even attempt to keep up with it all? And for the (lack of) quality the new inference engines provide (as tradeoff to speed / resource use), is it really worth using them?
submitted by /u/Kahvana [link] [comments]
#llama.cpp#Ollama#KoboldCpp#vLLM#ExLlama3#TabbyAPI#LiteRT#Qwen3.8-Flash-Next#DeepSeek-v4.1-Flash-1017