NVIDIA unveiled Reshard, a communication method for moving updated reinforcement learning model weights from training GPUs to inference GPUs. Teams using separate GPUs can reduce update waits by sending needed shards directly instead of gathering full weights first.

1. Gathering full weights before sending them again

Reinforcement learning post-training is additional training after pretraining. It uses the current policy to generate training data and periodically updates inference GPU weights. This update is called a weight refit. Training and inference have different compute patterns and weight layouts. Moving large amounts of data efficiently between the two GPU groups is therefore a challenge.

A typical existing method gathers shards from training GPUs to assemble full parameters, then sends them to inference GPUs. Inference GPUs retain only the portions they need, but receive the full parameters in the process.

Assuming BF16 weights at 2 bytes per parameter, NVIDIA estimated about 5.6TB for Kimi K3's 2.8 trillion parameters and about 4.8TB for Qwen 3.8 2.4T A95B's 2.4 trillion parameters. Logical data volumes for gathering and transfer combined, the sum of data sizes at each stage, are about 11.2TB and about 9.6TB, respectively.

2. Connecting required shards directly

Reshard describes source and destination weight layouts using DTensor, PyTorch's distributed tensor representation, and transfers only the required regions. To redistribute a row-sharded matrix into column shards, each GPU receives and assembles only the portions corresponding to its assigned columns.

NCCL M2N Reshard is an open-source, NCCL-based library implementing this approach. Teams describe both layouts and invoke the transfer, leaving the specific communication routes to the library.

3. Network transfers and exchanges within GPU groups

Destination GPUs sharing NVLink, a high-speed GPU interconnect, receive different portions over the network and exchange them when replicating the same data. The two transfers overlap, reducing repeated network transmissions of the same data to each replica.

4. Lower latency measured in NeMo RL

NVIDIA reported 2.7–5.7x faster refits with Reshard than with gather-and-transfer in NeMo RL measurements. The cases below use different hardware, models, and parallel layouts, comparing latency between the two methods within each case.

NVIDIA reported that Qwen 3 235B on DGX H100(NVL8) dropped from 6.4 seconds to 1.9 seconds. Training used tensor 4, pipeline 8, and expert 16 parallelism; inference used tensor 8 and data 16 parallelism. These distribute weight shards, consecutive model stages, expert groups, and replicas of the same model across GPUs, respectively.

DeepSeek v3 671B used pipeline 16 and expert 16 parallelism for training, and tensor 32 and data 8 parallelism for inference. NVIDIA reported a drop from 14.9 seconds to 2.6 seconds on DGX H100(NVL8), and from 6.1 seconds to 2.3 seconds on DGX GB200(NVL72) with this configuration.

NVIDIA reported that Nemotron Ultra v3 on DGX GB200(NVL72) dropped from 8.2 seconds to 2.3 seconds. Training used tensor 8, expert 32, and pipeline 2 parallelism; inference used tensor 8 and data 8 parallelism.

In a separate refit-only microbenchmark, a performance test isolating a specific task, NVIDIA reported a 4.3-second refit for the BF16 Qwen3.8 2.4T model. This figure was presented as a refit result for a trillion-parameter-scale model.

5. Deployment and shifting costs

NeMo RL uses Reshard for feedforward projection weights, which transform values in each layer, while keeping other weights on the existing gather-and-transfer path. NVIDIA says these weights account for about 95% of parameters in recent large language models. This focuses optimization on the largest data volumes while handling the varied shapes of other weights through the existing path.

The configuration is as follows.

policy:
  generation:
    colocated:
      enabled: false
    refit_transport: nccl_reshard

NeMo RL documentation marks this feature as experimental. Training and generation must run on separate GPUs, with Megatron used for training. Generation backends are limited to vLLM or Megatron, and generation-side pipeline parallelism is unsupported.

Reducing duplicate network transfers shifts part of the work of completing replicas to NVLink communication between destination GPUs. Quantized weights, which use reduced precision, handled by Reshard's bulk path are expanded to BF16 for transfer and requantized at the destination as needed. Format conversion costs therefore need consideration alongside transfer savings.

Reshard's value lies in replacing repeated transfers of full parameters with transfers of required shards to reduce update waits for large models.