CNCF discusses Kubernetes orchestration for agentic AI inference
Agentic AI is creating new demand for Kubernetes infrastructure as coding agents make repeated inference calls, Cloud Native Computing Foundation executive director Jonathan Bryce said during a sponsored theCUBE interview. Bryce said a coding agent can spawn 10 sub-agents, each of which may make thousands of inference calls over several hours. CNCF’s new AI inference and agentic track addresses related projects: the llm-d inference engine and Agent Substrate aim to improve the use of existing hardware, while OpenShell focuses on securing and controlling agents. The report also notes that OpenShell entered CNCF neutral governance following Nvidia’s contribution. The Linux Foundation sponsored the interview segment but said it did not control the editorial content.
Unlike early chatbots that triggered one inference call per message, agents repeatedly call models during a task. The report identifies model serving, hardware utilization, security and control as infrastructure requirements for these workloads, making orchestration a focus of CNCF’s agentic AI work.
The rise of agentic AI systems is giving Kubernetes, long valued for its reliability and predictability, an unexpected second act.
As AI spending shifts from training models toward serving them and running agents, the open-source community is developing tools to handle the growing volume of inference requests. The orchestration platform was built from the start for dynamic, distributed workloads, which gave fast-growing AI companies a reliable foundation, according to Jonathan Bryce (pictured), executive director of the Cloud Native Computing Foundation.
“The AI boom has sparked a massive investment and a huge build-out of infrastructure overall,” Bryce said. “Having Kubernetes built from the beginning to handle really dynamic distributed workloads as those companies were growing like crazy — they had a really reliable, solid foundation to build on at the same time.”
Bryce spoke with theCUBE’s Rebecca Knight at KubeCon + CloudNativeCon NA, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how CNCF projects can meet rising inference demand, make better use of data center hardware and secure AI agents. (* Disclosure below.)
#Cloud Native Computing Foundation#Kubernetes#llm-d#Agent Substrate#OpenShell#Nvidia