NVIDIA Agent Substrate: Idle session reuse, shared worker pools

1. Reducing the waste of assigning a worker to every session

Agents repeatedly run commands or edit files, then wait for model responses or human input. Files, processes, and working directories must remain intact until the next step. But keeping a Kubernetes pod, a task execution unit, running for every session consumes memory even while idle. Creating a new pod each time can take several seconds for scheduling, which selects a host computer, and for image and network preparation.

Coding assistants must preserve environments as they alternate between model calls and repository work. This calls for saving and resuming paused task state rather than assigning a dedicated execution environment to each of many sessions.

In reinforcement learning post-training, which further trains an already trained model, requests arrive in batches at CPU-focused environments such as code runners or simulators. NVIDIA says the scale can grow from hundreds of environments in early experiments to hundreds of thousands in production. The slowest environment determines when the next training batch is ready, so delays in creating environments leave GPUs waiting.

2. Saving idle state and restoring it on a free worker

Agent Substrate places a separate control layer for resource allocation and state management above a set of ready Kubernetes worker pods, connecting many agent environments to a smaller worker pool. Sandboxes, isolated execution spaces, use gVisor or micro virtual machines, small computer environments isolated through virtualization. When an environment becomes idle, its memory and filesystem state are saved and its worker is released. On the next call, work resumes from the saved state on a free worker.

Restoration does not require creating a new pod, and workers themselves are ordinary Kubernetes pods, so cluster scheduling and quota policies still apply. According to the company, the runtime is designed to run millions of agents and tools at 10x the density of standard containers.

3. Handling waits at capacity and resource contention within sandboxes

A feature was developed to avoid immediately failing new requests when all workers are busy. Instead, the router, which forwards requests to appropriate execution targets, holds requests within a set limit and attempts to resume them when a worker becomes free. Follow-up improvements were also introduced to keep requests for already running environments from getting stuck in the queue, an ordered list of requests awaiting processing.

Sandboxes can include helper containers for logging, proxying, which forwards requests on behalf of other services, and metric collection. Applying per-container CPU and memory limits in micro virtual machine sandboxes allows only the container exceeding its memory limit to stop while the rest of the environment keeps running.

A design was also proposed to suspend and revive GPU sandboxes where researchers compile code or perform computations. If device providers supply safe suspension methods and the device characteristics needed for restoration, the runtime saves them and restores the state on a compatible device. NVIDIA GPUs are the first supported target.

4. Feeding inference state back into environment management

While Agent Substrate manages environments where agents use tools, NVIDIA Dynamo coordinates model inference, the process in which trained models produce results on GPUs. The Dynamo router tracks the location of KV caches containing computation results from previous model calls and the congestion of GPU workers. It can also use session identifiers, values that distinguish sessions, and tool-call boundaries to coordinate subsequent model requests.

The core of the integration is feeding inference progress information back into agent environment management. Dynamo can provide signals such as estimated time to first token, request progress, cache readiness, and GPU pool saturation, the extent to which available resources are in use. Agent Substrate is designed to use these signals to decide whether to keep environments active, save them during long inference waits, or bring them back as model responses approach. The approach aims to reduce both the cost of always-on environments and the delay of restarting them every time.

5. Efficiency gains require coordination across layers

Preventing idle sessions from occupying workers and reusing state can reduce CPU and memory waste and the burden of restarting at each step. In return, operators must coordinate signals and resource operations across the two layers managing environment execution and inference. As the cost of keeping resources active falls, coordinating state preservation and resumption timing becomes more important.

GPU state restoration generally requires the same GPU model and sometimes the same driver version; without a compatible device, execution starts afresh. NVIDIA said the APIs are still changing, GPU support is being rebuilt on a new Kubernetes resource allocation foundation, and some Dynamo integrations remain experimental.

Agent Substrate is an independent open-source project donated to the Cloud Native Computing Foundation (CNCF) as a sandbox project. NVIDIA Dynamo handles inference, KAI Scheduler places worker pods, and Agent Substrate manages stateful agent environments. Because worker pods are ordinary Kubernetes pods, they can be placed alongside GPU trainers or inference servers.

Agent Substrate's value lies in linking task state preservation with inference progress, turning agents' waiting time into an opportunity to reuse shared resources.