NVIDIA introduced Dynamo's architecture for generating training responses and operating inference workers for reinforcement learning. Dynamo is a serving layer, the software layer that handles model requests. It distributes generation requests across workers, reduces repeated input computation and weight update overhead, and reroutes requests when workers fail. Teams building reinforcement learning systems can operate generation infrastructure separately while retaining their existing training frameworks.
1. Response generation speed determines training progress
Reinforcement learning improves models based on rewards. Inference workers generate data linking model outputs, actions, and interactions with environments. These generated results are called rollouts. The training framework evaluates them, updates weights, and sends those weights to workers. How quickly the required data is collected and new weights are distributed therefore affects the next training step.
Actual worker load varies by task. A reasoning problem may end with a single answer, while a coding agent explores repositories, edits files, calls tools, and runs tests. Even with equal request counts, completion times can differ. When training waits for a batch of samples, the last task delays the start of the next step.
Multiple attempts at the same task may share prefixes, such as instructions or repository information. Recomputing these inputs consumes GPU resources before new responses are generated.
New weights produced during training must reach workers when the algorithm requires them. NVIDIA identified three challenges: completing required samples sooner, reducing repeated prefix computation, and delivering new weights on time.
2. Separating training frameworks from inference operations
Dynamo sits between the training framework coordinating rollouts and inference workers, routing requests to suitable processing capacity. Inference engines perform the actual model generation. The training framework manages optimization, rewards, environments, checkpoints storing training state, and sample selection for training.
Dynamo handles routing, worker management, and serving control while monitoring worker status. ModelExpress distributes model weights and artifacts through supported integrations. Generation requests and weight updates follow separate paths. The framework selects update targets and controls update timing and sample acceptance. Training rules therefore remain with the framework even when serving operations are separated.
3. Considering both caches and worker load
When processing the same input repeatedly, routing it to a worker that already holds it can reduce prefix recomputation. However, always choosing cached workers can lengthen queues and delay overall completion. Dynamo's router considers both worker load and the reuse benefits of KV caches, which store computed keys and values, to reduce duplicate computation and waiting.
For weight distribution, ModelExpress reuses weights already in the cluster to reduce reloads from remote storage. Compatible workers can use NIXL to copy weights between GPU memory. GPU-to-GPU copying uses weights and communication resources within the cluster.
Delta transfers are also supported, compressing and sending updated portions instead of the entire model. This can reduce transfer volume, but workers need the corresponding baseline weights to construct the new state. The training framework continues to manage update timing and sample validity.
4. Handling worker failures and expanding real-world use
When Dynamo detects a lost worker, it routes new requests to other available workers and adds replacement workers to routing once ready. Replacement workers can load a checkpoint and apply subsequent changes to catch up with training progress. Supported configurations also allow moving in-flight requests or gracefully draining workers.
NVIDIA's examples included Microsoft AI's post-training rollout inference, Cognition's SWE-1.7 post-training worker management and routing, and Prime Intellect's distributed inference coordination. CoreWeave reported up to 15x faster weight updates using Dynamo and ModelExpress. H Company reported reducing average training step time by up to 20% and GPU usage time by up to 5% using both tools. These reported gains measure different things: 15x refers to weight update speed, 20% to training step time, and 5% to GPU resource usage. Priorities for load balancing, cache reuse, and distribution optimization depend on whether required samples finish late, repeated input computation is high, or weight distribution is slow.
The listed open-source integration targets were verl, NeMo RL, Slime, and prime-rl. The approach connects these systems to the inference serving layer rather than replacing existing reinforcement learning frameworks.
Dynamo's core approach leaves training decisions with the framework while coordinating repeated generation, weight distribution, and worker recovery in the serving layer.
