Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

llm-d 1.0 adds enterprise controls for self-hosted inference

AI summary

IBM and other contributors released version 1.0 of the open-source llm-d distributed AI inference framework, adding multi-tenant quality of service, zero-downtime upgrades and agent traffic controls for Kubernetes-based model serving. The release also integrates context-cache reuse across multi-turn requests, tiered cache storage, batching optimization and distributed deployment of large MoE models. IBM and Red Hat said a demonstration ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at five to 10 times lower cost than a hosted service.

In-depth insight

llm-d 1.0 consolidates multi-tenant management, uninterrupted upgrades and agent-request prioritization into an open-source inference framework, giving enterprises a new version for operating self-hosted models with greater control.

IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure.

The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.

Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are its lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.

The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform.

“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, wrote in e-mailed comments. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”

#llm-d 1.0#IBM#Red Hat#Google#CoreWeave#Nvidia#Advanced Micro Devices#Cisco#Hugging Face#Intel#Mistral AI#vLLM

Original · 4 min readRead the original
Group chat

No comments yet. Start the conversation.