创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

llm-d 1.0发布,强化企业自托管推理

AI 总结

IBM与llm-d其他贡献者发布开源分布式AI推理框架llm-d 1.0,为Kubernetes上的企业自托管模型服务增加多租户服务质量、零停机升级及智能体流量控制。该版本还整合跨多轮请求的上下文缓存复用、缓存分层存储、批处理优化和大型MoE模型分布式部署。IBM与Red Hat称,其演示在自托管基础设施上同时运行3000个软件工程智能体,成本比托管服务低5至10倍。

深度见解

llm-d 1.0把多租户管理、无中断升级和智能体请求优先级等生产级能力整合进开源推理框架,为企业控制自托管模型的可靠性与基础设施成本提供了新版本。

IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure.

The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.

Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are its lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.

The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform.

“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, wrote in e-mailed comments. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”

#llm-d 1.0#IBM#Red Hat#Google#CoreWeave#Nvidia#Advanced Micro Devices#Cisco#Hugging Face#Intel#Mistral AI#vLLM

原文约 4 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。