Agent Substrate 1.0: Stateful execution, isolated sandboxes

Agents retain state even while waiting

Agents perform tasks, then wait for model inference, external services, or human review. According to Google's announcement, waiting accounts for 90~95% of interactive sessions. Even without CPU use, the language runtime, local storage, files being edited, and tool sessions must remain in memory and on disk.

Keeping a dedicated Pod running enables immediate responses but occupies compute resources even while idle. Google says scaling serverless containers to 0 can require recovery steps such as cloning storage and reinstalling packages. Waking up can then take more than 20 seconds. Google's goal is to resume in under 500 milliseconds while retaining memory and disk state and eliminating idle Pod costs.

Training and evaluation face another bottleneck. Reinforcement learning loops that improve behavior through rewards, or benchmarks comparing performance, can run agent code across thousands of tasks at once. Google says creating and deleting a Kubernetes Pod for each task can cause request bursts. These can strain etcd, the distributed store for cluster configuration, and kube-apiserver. They can also exhaust IP addresses in subnets, logical networks that divide IP address ranges.

Separating state from compute enables reuse

Agent Substrate separates Actors, which hold agent session state, from Workers, which perform computation. An Actor contains the file system, environment variables, memory state, and network rules. Workers provide compute resources from a shared pool of prepared Kubernetes Pods. When an agent is idle, its Worker slot can be freed and reused for other tasks.

When a request reaches a stopped Actor, a forwarding proxy holds it in memory and notifies the controller that reconciles the desired state. The controller assigns a free Worker slot. It forwards the request once state is restored from tiered storage combining storage of different speeds. When the task ends, it saves state again and releases the slot. Google says this enables agents to activate, stop, and resume without creating a new Pod object each time.

This design aims to reduce compute resources tied up in idle Pods and the Pod creation burden on the control plane that manages cluster operations. However, sharing compute requires operating a warm Worker pool and allocating storage for stopped session state. Costs and operational work therefore shift from idle execution to pool sizing and state storage rather than disappearing.

gVisor creates execution boundaries and helps restore state

Agents can run model-generated code, making host isolation important. Ordinary Linux containers share the host kernel. gVisor, also donated by Google, is a userspace application kernel written in Go. It intercepts system calls requesting operating system functions in userspace, where applications run outside the kernel, separating untrusted code from the host kernel.

gVisor manages its own memory and process table containing information about running programs. This enables pausing an execution environment, saving memory pages, and restoring them on another node. According to Google's announcement, Agent Substrate restores state in under 500 milliseconds. Google also says the same interface supports small, fast-starting Cloud Hypervisor-based micro virtual machines. These serve tasks requiring hardware virtualization, which creates virtual computers, or custom kernel modules, which add features to the operating system core.

The CNCF donation opens a foundation for testing

Google Cloud announced the donation of Agent Substrate and gVisor to the CNCF Sandbox stage in Salt Lake City, host of KubeCon North America. OpenRL was also mentioned. In Google's proposed setup, gVisor handles isolation, Agent Substrate handles agent execution, and OpenRL coordinates training and evaluation. Agent Substrate has released version 1.0 and provides code and documentation for testing.

Developers can find guides and architecture documentation at ate.dev and code in the agent-substrate/substrate repository on GitHub. The substrate-dev and substrate-user channels on CNCF Slack also offer ways to join the community. Google highlighted a KubeCon workshop for testing Worker pool deployment and restoration from snapshots that capture state at a specific point in time. It also highlighted a demo running multiple agents in a pool on GKE.

The donation combines agent state preservation and secure execution on an open Kubernetes foundation, shifting operational focus to warm resources and state storage management.