Buoyant 推出开源 vLLM 控制平面 Kinference
Buoyant 宣布推出 Kinference,这是一个采用 Apache 2.0 许可证、用于基于 vLLM 和 Kubernetes 构建推理平台的开源控制平面。它提供令牌限流、租户用量与成本跟踪、推理专用负载均衡及策略路由,并通过兼容 OpenAI 和 Anthropic 客户端的 HTTP 端点接收推理流量。在 Buoyant 的公开基准测试中,Kinference 在一种工作负载下的首令牌时间和端到端延迟中位数低于标准 Kubernetes 负载均衡,其共享前缀工作负载的 goodput 达到 7.4 倍,多轮对话工作负载约为 3.5 倍。
Kinference 为运行开放权重模型提供了一个开源控制层,包含策略控制、按租户统计、成本跟踪,以及结合 GPU 队列长度、历史性能和 KV 缓存内容进行的负载均衡。
We’re excited to announce Kinference [kay-inference], a new open source project for building scalable inference platforms with vLLM.
We created Kinference at Buoyant after watching our customers struggle to run open weight models in their environments. We designed Kinference to be the control plane layer for the platform you need to build. It uses vLLM to do the underlying inference, and provides the set of features necessary for building an inference platform in practice. This includes token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.
