CUDA Toolkit 13.5: Full Vera Rubin support, GPU resource management

1. Direct control over GPU resource allocation

The release expands how GPU compute units called streaming multiprocessors (SMs) are assigned to locality domains, which group nearby memory access. Defaults give each domain equal SM counts and leave spare SMs outside. Combining the new settings can include spare SMs in selected domains, but SM counts may then differ across domains.

Applications can query resource partitions to check each domain's SM count and distribute work. They can thus keep domains equally sized for predictability or use more of the available SMs in each domain.

MPS v3, which lets multiple processes share a GPU, adds weighted time slicing. A multiplier on each server's base time adjusts compute time, allowing more time for servers handling jobs that need longer computation.

SM partition load balancing lets partitions borrow SMs outside their base partition when capacity is available. Partitions without clients can lend resources. Borrowing and lending are configured separately, and MPS reclaims borrowed resources before reassigning them.

Where memory usage notifications are unreliable, polling can read current usage and limits directly to decide whether to reclaim resources. This reduces reliance on notification delivery but adds work to each reclamation decision.

2. Multi-GPU communication in CUDA Tile

CUDA Tile kernels, code that runs on GPUs, can now communicate directly with multiple GPUs. Single-target pointers access memory on GPUs linked by NVLink, an inter-GPU connection technology. Multi-target pointers and dedicated operations combine results or store data across GPUs on an NVSwitch fabric, a switch network linking multiple GPUs. The fabric must support NVLink SHARP, a communication technology that combines results from multiple GPUs.

Preparing shared memory mappings before execution lets GPUs see the same memory and combines communication with computation inside a kernel. For example, one kernel can combine matrix multiplication with summing results across GPUs.

Calling library functions from the existing thread-based SIMT execution model inside Tile kernels is also in public preview. Instead of passing data between separate kernels, kernels can directly process data they already hold. Support for the relevant libraries will be enabled progressively.

3. Development conveniences in Python and CCCL

Batch copying in CUDA Python core libraries 1.2 submits multiple buffer copies in one driver call. CUDA Graph, which reuses execution workflows, can update kernel parameters and memory copies without rebuilding the graph. Applications can thus reuse existing graphs when their parameters change.

Programmatic dependent launch lets cooperating kernels overlap in the same stream on GPUs with compute capability 9.0 or higher, a rating of GPU instruction features. A following kernel can start independent work before the preceding kernel finishes.

Below is the configuration example from the original article. The kernel must implement the synchronization protocol for programmatic dependent launch.

from cuda.core import LaunchConfig, launch

config = LaunchConfig(
    grid=num_blocks,
    block=256,
    programmatic_stream_serialization=True,
)

# The kernel must implement the PDL synchronization protocol.
launch(stream, config, dependent_kernel, *kernel_args)

In CCCL 3.5, a collection of GPU algorithm libraries, CUB's public tuning policies let developers directly adjust algorithm execution settings such as threads per block.

Floating-point accumulation can request a fixed summation order to reproduce results across repeated runs with the same GPU, inputs, and build configuration.

Batched selection, which picks the top or bottom K items in each of multiple segments, is also new. NVIDIA said this feature has been integrated into FlashInfer, a language model inference library.

4. Performance improvements in math and image libraries

Improvements to cuBLAS symmetric updates using native 64-bit floating-point (FP64) arithmetic apply to scientific computing such as eigensolvers and matrix factorizations. NVIDIA reports up to 1.5x speedups for some operations on NVIDIA B200, with no code changes needed to benefit.

cuSPARSE ALG3 for CSR (compressed sparse row) sparse-dense matrix multiplication improves FP32 (32-bit floating-point) and FP64 operations. NVIDIA reported 1.65–2.27x speedups for this operation on data center GPUs.

Table: cuSPARSE sparse triangular solve ALG2 performance relative to ALG1

Conditions: Based on NVIDIA's announcement. Performance compares ALG2 with ALG1 for sparse triangular solves; the original article did not specify the measurement hardware.

Unit: Speedup (x)

The geometric mean calculates an average by multiplying multiple ratios.

For repeated sparse triangular solves, solve time and setup time can be considered separately. ALG2 improves repeated solve performance. NVIDIA also reports a 2.79x geometric mean speedup for pre-solve analysis, reducing setup time.

NVIDIA said cuSPARSE graph coloring also achieved a geometric mean speedup of about 3x.

NVIDIA reported a 10–600% throughput increase for nvJPEG GPU hybrid decoding, depending on image size and whether standard Huffman tables are used.

5. Moving runtime overhead to deployment

nvprune precompilation converts intermediate device code, which is adapted to the execution device, into GPU machine code during packaging. This removes runtime compilation overhead for deployments with fixed target architectures. Deployers instead must choose architectures for target environments and validate the output on supported GPUs.

Windows deployments on Rubin and later GPUs must use WDDM or the compute-only MCDM driver mode instead of the existing TCC mode.

cuobjdump symbol output changes to an annotated four-column format, so scripts that parse it must also be updated.

CUDA 13.5 expands options for partitioning GPUs and connecting multiple GPUs, letting developers directly balance resource utilization against execution choices.