LMDeploy
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Olla v0.1.0 adds TEI support and reliability improvements
The Olla project released v0.1.0, its 25th release. The update adds Hugging Face Text Embeddings Inference support for embeddings, reranking and classification; configurable retry and failover behavior; a reworked dashboard; improved model-capability detection; and fixes for concurrency, request statistics, rate limiting, routing, backend discovery and configuration validation. Olla is a Go-based open-source proxy and load balancer that sits between applications and local inference servers. It supports multiple runtimes, including Ollama, vLLM, SGLang, llama.cpp and LM Studio, and provides OpenAI- and Anthropic-compatible interfaces with translation where required.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my LMDeploy experienceAsk about LMDeploy
Nobody has shared their experience with LMDeploy yet.