Olla v0.1.0 新增 TEI 支持并改进可靠性
AI 总结
Olla项目发布v0.1.0,这是其第25个版本。此次更新加入Hugging Face Text Embeddings Inference支持,可处理嵌入、重排和分类;新增可配置的重试与故障转移行为;重做仪表盘;改进模型能力检测;并修复并发、请求统计、速率限制、路由、后端发现和配置验证等问题。Olla是一款基于Go的开源代理和负载均衡器,部署在应用程序与本地推理服务器之间。它支持包括Ollama、vLLM、SGLang、llama.cpp和LM Studio在内的多种运行时,并提供OpenAI和Anthropic兼容接口,必要时可进行协议转换。
深度见解
据项目介绍,该版本为混合本地推理环境提供了一种轻量的统一管理方式,可集中处理模型感知路由、负载均衡、故障转移、健康检查、可观测性和API转换。Olla可作为单个Go二进制文件或Docker容器运行,无需Kubernetes、数据库或外部服务。
| Hey folks, About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work. We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!). The problems we're trying to solve haven't really changed and now some of our customers use it: That's where Olla comes in. It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers. Some of what it does today: What's new in v0.1.0? This release is largely about improving reliability and making Olla more useful for people running mixed inference environments. We've also had some great contributions from the community, which has been encouraging to see. Why build another LLM proxy? We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started. We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model. It's not trying to replace inference engines. Just make running a bunch of them together less painful. Links: If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today. And as always, if you give Olla a spin, let us know how it goes (and what breaks!). submitted by /u/2shanigans [link] [comments] |
#Olla#Olla v0.1.0#Hugging Face Text Embeddings Inference#Ollama#vLLM#SGLang#llama.cpp#LM Studio#MLX#Docker Model Runner#LMDeploy#Lemonade