llm-d 1.0 adds production inference features for agent workloads
llm-d 1.0 adds multi-tenant resource management, request-priority scheduling, high availability, rolling upgrades and context-aware routing for agent workflows. An IBM Research benchmark deployed an approximately 753-billion-parameter model through llm-d, supporting up to 3,000 concurrent coding agents at peak output throughput above 6.6 million tokens per minute. Red Hat AI 3.5 also added governance capabilities, while IBM and Red Hat jointly disclosed that Lightwell Clearinghouse had reached general availability.
The release is intended to address production requirements for large-scale inference, including concurrent agent workloads, multi-tenant resource management and operational reliability. The report also notes that the benefits of these capabilities in real production environments still require further validation.

“刚选定模型、完成平台搭建,业务流程也已经接入。但没过几天,市场上又出现一个表现更好的模型,CEO转头就问CIO:能不能也试试?”红帽全球副总裁兼大中华区总裁曹衡康(Victor Tsao)提到了企业落地AI时经常遇到的一个困境。
对企业来说,换模型并不像个人更换一个聊天工具那么简单。模型背后还连接着GPU适配、推理服务、数据接口和业务流程,前一套系统刚刚跑顺,新模型就来了,企业既不想错过技术进步,也不愿意为此重新搭建整套环境。
Agent的出现又带来了另一层压力。当智能体开始接入企业内部系统,能够访问什么数据、执行哪些操作,就成了必须明确的问题。如果缺乏有效的权限约束,Agent为了完成任务,甚至可能直接删除生产数据库。
#llm-d#Red Hat AI 3.5#Lightwell#Lightwell Clearinghouse#IBM Research#Red Hat#IBM#NVIDIA#Intel#AMD#MetaX#EvalHub