CUDA 13.4 Brings Memory-Enforced GPU Sharing to Kubernetes
NVIDIA Technical Blog says CUDA 13.4 adds transparent, application-aware memory enforcement for GPU time-sharing while retaining the SM-sharing compute mode available through Multi-Process Service. Workloads can request an exact memory allocation or a guaranteed minimum with optional burst capacity, and the new open-source GPU Fractioning Operator applies those limits and compute modes in Kubernetes through pod annotations. NVIDIA also says KAI Scheduler v0.18.0 can select GPU fractions and write the required annotations, though no independent test results were supplied.
The release provides a vendor-reported mechanism for running multiple AI workloads on shared GPUs with explicit memory and compute limits, directly relevant to GPU utilization and cluster orchestration.
Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads with the infrastructure they already have.
NVIDIA GPUs already support several approaches to sharing. Multi-instance GPU (MIG) splits a GPU into hardware-isolated instances that are transparent to the application, time-slicing lets several processes take turns without memory enforcement, and Multi-Process Service (MPS) lets them run concurrently in separate address spaces (sm-sharing).
CUDA 13.4 now enforces a memory boundary that is transparent to the application with a new time-sharing mode, alongside the compute sharing mode (sm-sharing) that already existed in MPS.
Here is what the CUDA release brings:
- An exact memory amount: A workload asks for a memory size and CUDA holds it there. An allocation above that size fails, just as it would on a smaller GPU.
- A range: A workload gets a guaranteed minimum plus a maximum it can grow into when memory is free.
- A compute mode: Each workload can choose to use its compute in an sm-sharing mode rather than using the default time-sharing mode.
- A transparent memory limit: Each workload sees only a fraction of the GPU memory.
#CUDA 13.4#NVIDIA#GPU Fractioning Operator#KAI Scheduler#NVIDIA GPU Operator#Multi-Process Service#Kubernetes