PyTorch 2.15: New communication defaults, platform performance improvements

1. GPU communication switches to a new implementation

The NCCL backend, the communication layer exchanging data between NVIDIA GPUs, moves to a new implementation. Specifying nccl selects it, and GPU communication channels are created when the communication group is initialized.

New logging helps diagnose communication hangs. If one training process times out, all processes write their current work state to disk, allowing inspection of what they were doing when the hang occurred.

Below is the initialization example from the official release notes. Alongside the default implementation, it shows options for independent per-peer communication channels and the legacy implementation.

dist.init_process_group("nccl", device_id=torch.device("cuda", local_rank))  # new implementation
dist.init_process_group("nccl-lazy", device_id=torch.device("cuda", local_rank))  # independent P2P communicators
dist.init_process_group("nccl-legacy")  # legacy implementation, removed in the next release

2. Control training precision and memory allocation timing

FSDP2, which distributes model state across GPUs, gains finer precision controls. Sensitive parameters such as router gates or normalization weights can stay in 32-bit precision while the rest of the model trains at lower precision.

With reduce_dtype=None, gradients accumulate and reduce according to each parameter's gradient precision setting. The release notes' example uses 32-bit parameters and bfloat16 model computation. Gradient processing changes from 16-bit to 32-bit, doubling the data volume for reduce-scatter communication. This preserves higher gradient precision at the cost of more communication. Explicitly set the reduction precision to retain the previous behavior.

Pipeline parallelism, which splits model execution into stages, allocates buffers just before receiving and frees outputs after sending. Allocating buffers only when needed reduces peak memory use. New controls also overlap gradient reduction with subsequent computation.

Distributed communication adds a one-sided transfer API that connects to peers through a store without communication groups or collectives. Its first backend is NIXL. Transfers can be placed on a GPU execution stream and captured in CUDA graphs that record and replay sequences of work.

3. Accelerate text generation operations on Apple silicon

MPS, the GPU execution layer on Apple silicon, moves more operations to native Metal kernels. Replacing operations that used an intermediate execution layer with direct kernels reduces dispatch overhead.

The official announcement reports that softmax, converting scores for 128,256 vocabulary entries in one row into probabilities, takes one eighth of the time. Upsampling backward passes are up to 21.7x faster. Norm operations measuring the magnitude of an entire tensor are also reported to be over 100x faster.

The announcement reports a 1.23x geometric-mean speedup for F.linear during text-generation decoding, reaching 5x for small projections. The geometric mean is an average calculated using multiplication. For decoding in models that select subsets of subnetworks, such as mixture-of-experts models, a new grouped matrix operation path on M3 Ultra raises throughput by 1.32–1.34x.

In separate linear algebra results, the announcement reports that QR decomposition of a 512×512 matrix drops from about 7 seconds to 1.5 milliseconds. QR decomposition splits a matrix into an orthogonal matrix and a triangular matrix.

4. Choose installation packages and runtime environments together

The default installation wheel switches to CUDA 13.2, and CUDA 13.4 wheels are also available. Both require NVIDIA drivers 580 or later and Turing-generation or newer GPUs. For environments that must retain CUDA 12.6 or 13.0 wheels, the official release notes recommend using an earlier version.

Supported Python versions are 3.11–3.15. Upgrading from a 3.10 environment also requires upgrading Python. MPS acceleration on Apple silicon requires macOS 15 or later.

ROCm 10.0 wheels are added for AMD. ROCm wheels now use TheRock packages, installing libraries as dependencies rather than bundling them directly.

5. Prepare to migrate existing training setups and stored data

The new communication backend allocates GPU memory even for unused groups and binds each group to one device. Setups where one process uses multiple GPUs therefore need the legacy implementation. That implementation and the environment-variable fallback will be removed in the next release.

Distributed checkpoints saved in 2.15 cannot be read by 2.14 or earlier, so rollback plans must account for stored-data compatibility.

PyTorch 2.15 offers faster operations and broader training controls, but migration planning must also cover communication setups and stored data.