llm-d 1.0发布,强化企业自托管推理
IBM与llm-d其他贡献者发布开源分布式AI推理框架llm-d 1.0,为Kubernetes上的企业自托管模型服务增加多租户服务质量、零停机升级及智能体流量控制。该版本还整合跨多轮请求的上下文缓存复用、缓存分层存储、批处理优化和大型MoE模型分布式部署。IBM与Red Hat称,其演示在自托管基础设施上同时运行3000个软件工程智能体,成本比托管服务低5至10倍。
llm-d 1.0把多租户管理、无中断升级和智能体请求优先级等生产级能力整合进开源推理框架,为企业控制自托管模型的可靠性与基础设施成本提供了新版本。
IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure.
The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.
Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are its lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.
The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform.
“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, wrote in e-mailed comments. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”
#llm-d 1.0#IBM#Red Hat#Google#CoreWeave#Nvidia#Advanced Micro Devices#Cisco#Hugging Face#Intel#Mistral AI#vLLM