DeepSpeed has released a version improving memory handling for mixture-of-experts models, which select a subset of experts, and training across devices. Developers can use expanded parallelism and fixes. Code using removed or deprecated features needs review.
1. Sharing temporary storage across expert layers
In this version, released on October 20, AutoEP lets multiple layers of a mixture-of-experts model share one DeepEP buffer. Temporary storage previously allocated separately to each layer is now shared.
AutoEP's default combine operation also eliminates redundant 32-bit floating-point copies. Changes also overlap split planning, token packing, and input gathering. Both memory handling and operation scheduling have been revised.
Fused execution of some expert operations and a memory reclamation policy for recovering used memory were added as options. Developers can choose to enable these new paths.
2. Expanding training paths that partition the vocabulary
AutoTP adds support for vocabulary-parallel language model output heads and embeddings. Vocabulary partitioning now spans embeddings that represent inputs through to the output head, with improved support for models that share embeddings.
A fused operation for computing training loss with a partitioned vocabulary was also added. Extending both output head support and loss computation broadens the scope of processing that can use parallel training.
The release notes mainly consist of short titles for individual changes and include no performance comparisons or measurement conditions.
3. Fixing omitted training calculations and storage errors
ZeRO-1 and ZeRO-2 fix silent omissions of some groups when summing noncontiguous CPU gradients, the slope values computed during training. This corrects an error that left some training calculation results out of the sum.
ZeRO-2 now preserves 32-bit gradient accumulation precision during partitioned storage. This fix addresses accuracy when combining and storing training calculations distributed across devices.
Checkpoint conversion, which converts saved training states, now rejects maps that direct multiple pieces to the same storage shard address. This blocks layouts that would cause overlapping writes to the same location.
4. Check removed features and rejected settings first
Support for elastic training, which handles changes in device count, and progressive layer dropping, which removes layers during training, has been removed. Two older loss functions were also removed. Remaining compression library features and legacy curriculum learning, which starts with easier tasks, are deprecated. When upgrading, first check whether dependencies have been removed or merely deprecated.
Configuration documentation now explains sub_group_size and corrects its default to 1e9. This is worth checking when comparing existing configurations with documented defaults.
DeepSpeedCPUAdam now rejects the amsgrad setting, which it previously ignored. Projects retaining this setting should review their optimizer configuration before adopting the new version.
Expanded parallelism and calculation fixes offer benefits, while updating code using older features and removing disallowed settings impose migration costs. With features removed, changes to the features in use matter more to adoption decisions than the patch-release label.
5. Accelerator support and build requirements also change
NPU support now includes pinned host memory registration, which pins CPU memory for device access, and graph operations, which execute operations grouped into graphs. These changes strengthen support for device operations and memory handling.
NPU and XPU operation builders, tools that generate and build operation code, include changes to the build language standard.
Builds now use C++20 with torch 2.12 or later. Developers building operations in those environments should check this change.
On NPU, a deprecated Adam call has been replaced with a fused operation. The release identifies the following existing call as the replacement target.
npu_apply_adam(_w)This release broadens expert-model and parallel-training capabilities while leaving older-feature cleanup and accelerator-specific build checks as upgrade tasks.