Olla v0.1.0 adds TEI support and reliability improvements
The Olla project released v0.1.0, its 25th release. The update adds Hugging Face Text Embeddings Inference support for embeddings, reranking and classification; configurable retry and failover behavior; a reworked dashboard; improved model-capability detection; and fixes for concurrency, request statistics, rate limiting, routing, backend discovery and configuration validation. Olla is a Go-based open-source proxy and load balancer that sits between applications and local inference servers. It supports multiple runtimes, including Ollama, vLLM, SGLang, llama.cpp and LM Studio, and provides OpenAI- and Anthropic-compatible interfaces with translation where required.
The release gives operators of mixed local inference environments a lightweight way to manage model-aware routing, load balancing, failover, health checks, observability and API translation. Olla can run as a single Go binary or Docker container without requiring Kubernetes, a database or external services, according to the project.
| Hey folks, About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work. We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!). The problems we're trying to solve haven't really changed and now some of our customers use it: That's where Olla comes in. It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers. Some of what it does today: What's new in v0.1.0? This release is largely about improving reliability and making Olla more useful for people running mixed inference environments. We've also had some great contributions from the community, which has been encouraging to see. Why build another LLM proxy? We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started. We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model. It's not trying to replace inference engines. Just make running a bunch of them together less painful. Links: If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today. And as always, if you give Olla a spin, let us know how it goes (and what breaks!). submitted by /u/2shanigans [link] [comments] |
#Olla#Olla v0.1.0#Hugging Face Text Embeddings Inference#Ollama#vLLM#SGLang#llama.cpp#LM Studio#MLX#Docker Model Runner#LMDeploy#Lemonade