CUDA 13.4: GPU memory boundaries, scheduler integration

1. CUDA 13.4 enforces memory shares

GPU sharing options include MIG with hardware-isolated instances, time-slicing with processes taking turns, and MPS with concurrent execution in separate address spaces. CUDA 13.4 memory boundary enforcement lets shared workloads see only their configured memory allowance. If the limit is omitted and fixed at the request, larger allocations fail as they would on a smaller GPU.

Workloads can request a fixed memory size or specify both a guaranteed minimum and a maximum they can grow to when spare memory is available. Compute options are default time-slicing or MPS sm-sharing. With time-slicing, workloads take turns using the GPU; sm-sharing divides streaming multiprocessors to run workloads together.

2. Adding a component to automate node configuration

Enforcing memory boundaries in Kubernetes requires linking scheduler placement with GPU node configuration. Nodes must apply per-container memory limits and compute modes before processes start.

The component handling this, gpu-fractioning, is an open-source Kubernetes operator that automates node configuration. Released by the KAI Scheduler organization, it configures and runs MPS on each node and applies each workload's memory limits and compute mode.

It works alongside NVIDIA GPU Operator, which configures drivers and the container runtime environment. Usage is measured per workload share rather than for the whole GPU, letting operations teams monitor each workload's memory and compute usage.

3. Passing scheduler-assigned shares to containers

The scheduler selects a GPU with enough free memory, then records per-container assignments in Pod annotations. These annotations convey memory allocations, compute modes, and GPU devices as configuration notes. The operator reads and applies them on nodes, allowing integration with any Kubernetes scheduler that can write this information.

Developers request memory as an absolute size or a fraction of a GPU. The request is the guaranteed minimum share; a separate limit allows growth up to that limit when spare memory is available. The limit is optional; omitting it caps memory at the requested size.

4. KAI Scheduler enables integration through installation settings

KAI Scheduler is the first scheduler integrated with gpu-fractioning. It provides the new GPU sharing mode starting with version 0.18.0.

This mode, NvFractions, assigns per-GPU shares and passes them to the operator through annotations. Existing sharing modes remain available, and administrators choose a mode for each cluster. Using the following installation setting also installs gpu-fractioning as a dependency.

helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  -n kai-scheduler --create-namespace \
  --set global.nvFractions.set=true

5. The trade-off between time-slicing and concurrent execution

With time-slicing, workloads take turns using the entire GPU and are isolated so that one workload's errors do not spread to others. With sm-sharing, compute units are distributed among workloads as needed for concurrent execution. According to NVIDIA, when all n shared workloads are busy, both modes give each roughly 1/n of the compute capacity, while other workloads can use spare resources.

Concurrent workloads share a fault domain, so grouping mutually trusted workloads is recommended. Only one compute mode applies to a GPU at a time. KAI Scheduler tracks this per GPU, allowing the two modes on different GPUs within the same node.

The benefit is being able to colocate workloads that need only part of a GPU and monitor each share's usage. In return, operations teams must manage the node operator and MPS and configure the scheduler to pass assignment information correctly. Using spare resources through sharing adds the task of managing placement and node configuration together.

CUDA memory boundary enforcement and scheduler integration extend GPU sharing beyond simple concurrent use to managing per-workload resource shares.