DeepSpeed has released version 0.20.0 for model training developers. Users can opt into an execution policy for reproducible results under the same conditions. Operation capture and replay now support more accelerators. The release also fixes errors in preserving computed values during training and restoring saved training states.

1. Reproducibility arrives as an opt-in feature

This version adds an opt-in deterministic execution policy. Deterministic execution aims to reproduce results when computations repeat under the same conditions. Developers comparing training results gain a new execution option.

The release also includes fixes for training computation accuracy. In models that divide sub-networks into experts, gradient clipping now counts each expert's values only once. The norm used here is computed in 32-bit floating point.

It also fixes per-expert gradient scaling errors in configurations combining AutoEP and AutoTP, which partition experts and tensors. Alongside the new execution policy, the actual training computation paths have been revised.

2. Graph capture expands to non-CUDA accelerators

Hybrid Engine now supports graph capture on non-CUDA accelerators. Graph capture records an operation flow in advance for repeated reuse. This change expands the range of hardware that can use the feature.

Graph capture requires explicit enablement for each accelerator. Support has also been added for Kunlunxin XPU. Adoption reviews should therefore start by checking both accelerator support and enablement requirements.

Memory management has also changed to reflect the execution environment. The memory limit recognized by the CPU accelerator is capped by the process's resource limit. This distinguishes total system memory from memory allowed for an individual process.

3. ZeRO gradient preservation and memory handling are corrected

Stages 1 and 2 of ZeRO (ZeRO-1/2), which partition training states, preserve gradients across request boundaries until weights are updated. Gradients determine the direction of weight changes during training. This fixes the path that combines computations from multiple requests into one update.

Configurations that offload training states to the CPU use one accumulator per group and zero out unused gradients. ZeRO-2 now accumulates gradients across requests within the partition in this configuration. An accumulator is CPU storage that collects a group's gradients. A partition stores the corresponding share of the divided gradients.

ZeRO-3 copies off-device master weights used for updates to the device through a pinned memory buffer. Weight partitions already matching the master weights' data type are reused instead of copied separately. Both device transfer paths and copy management have been revised.

4. Training state restoration and storage errors are also fixed

Universal checkpoints are a storage format shared across training configurations. Loading them now restores training step values to the correct location for fused and graph-capture-compatible optimizers. These optimizers batch operations or support graph capture. An optimizer uses gradients to update weights.

CPU-resident optimizer states originally in 32-bit floating point retain that precision after saved states are loaded.

In ZeRO-3 configurations that keep training states on storage, a fix addresses parameter prefetch errors when swap buffers are already allocated. These buffers temporarily hold data during transfers between devices. DeepNVMe also fixes I/O failures with unaligned buffers.

Memory leaks in the engine and ZeRO optimizer have also been fixed. The fixes cover not only training computations but also state restoration, data movement, and memory management.

5. Optional settings offer flexibility but require validation

For ZeRO-1/2 training that combines gradients from multiple requests into one update, preservation across request boundaries is an adoption criterion. For ZeRO-3, configurations where weight partitions and master weights share a data type warrant review. The change reuses existing partitions instead of separate master-weight copies.

CPU training environments with process resource limits should be configured around the memory allowed for the process, not total system capacity. The change that reflects this restriction in the CPU accelerator's memory limit directly concerns these environments.

The release notes do not explain exactly how to enable the deterministic execution policy or which operations it covers.

The release's core is broader reproducibility options and accelerator support, alongside more stable ZeRO training and memory paths.