NVIDIA released Megatron Core 0.20.0 for teams training and serving large models across multiple devices. It offers finer training partitioning choices and an inference path that keeps output consistent when request batches change.

1. Finer control over distributed training configurations

The key addition is the experimental Megatron-FSDP v2 adapter, a module connecting features to existing training systems. FSDP distributes training state across devices. The adapter supports context, expert, and pipeline parallelism, plus hybrid data parallel sharding. Megatron Bridge integration was also added.

Hybrid FSDP allows different sharding levels within and outside groups connected by NVLink, an inter-device network. Dense sections, which compute across the model, and mixture-of-experts sections, which select experts, can also have separate settings. Sharding can thus be chosen for each hardware topology and model section.

Saving and restoring training state is also part of the integration. A path was added to save and load model and optimizer states using PyTorch distributed checkpointing. The optimizer updates weights during training, so this change covers both the model and the state needed to resume training.

2. Consistent MoE computation across changing request batches

Mixture-of-experts (MoE) inference now has batch-invariant execution. This path prevents results from varying with the other requests in a batch. Supported precisions are BF16, a 16-bit floating-point format, and MXFP8, a microscaling 8-bit number format. It covers vLLM fused expert operations, which combine operations, and expert matrix multiplication, which multiplies expert weights as matrices.

Per-request random seed handling was also strengthened. Requests with a specified starting value for random generation are insulated from batch composition, request migration, and other requests' random-number consumption. Alongside consistent expert computation, this matters for reproducing results in services that mix requests.

The inference engine also supports expert tensor parallelism. This splits tensor computation within experts across devices. These parallel execution options are separate from the training adapter, providing grounds to evaluate training and serving configurations independently.

3. Efficiency gains come with configuration review requirements

The new training adapter supports MXFP8 computation, quantized parameter gathering, mixed precision, and optimizers using multiple numeric precisions. Quantization represents numbers with fewer bits. More options target memory and compute efficiency, but adopting teams must align precision, parallelism, and state restoration paths.

This configuration review has specific constraints. The adapter's tensor parallel size and expert tensor parallel size must each be 1, and optimizer checkpointing currently excludes expert parallel layouts.

Supported dense model configurations can now capture and replay an entire training iteration as a CUDA graph. This groups GPU operations for replay. In inference, vLLM expert matrix multiplication buffers are retained for graph execution. The release notes also list company-reported performance improvements to the fused MoE path.

4. Fewer inference stalls from storage and connection handling

On the operational side, preparation of generated tokens for storage was moved outside the inference loop. This addresses the path where slow checkpoint writes stalled decoding. The focus is on continuing generation while handling result storage.

The HTTP frontend scales with the data parallel size. Tokenization, which splits text into model input units, was also moved outside the event loop coordinating requests. These changes improve processing flow between the request entry point and actual inference execution.

Generation for a request now stops when its client disconnects. This prevents remaining work from occupying processing capacity after a user leaves. While batch-invariant execution addresses output reproducibility, this change is more about managing service capacity.

This release gives training teams flexible sharding and helps inference teams maintain consistent service as request batches and operational flows change.