AICR for NVIDIA Dynamo: Validated runtime, pre-deployment checks

1. Aligning the foundation beneath the inference graph

NVIDIA Dynamo is a distributed inference framework that coordinates model serving across multiple GPU workers, or worker processes. Operating it requires GPU drivers, resource allocation, scheduling, networking, and monitoring to work together.

NVIDIA AI Cluster Runtime (AICR) is open source and provides this foundational software alongside Dynamo through recipes. A recipe is an assembly guide for the runtime, listing components to install, settings, and checks.

Each recipe pins component versions and compatibility requirements for a specific environment and workload. It also records settings, node requirements, and expected validation results to help reproduce the environment. NVIDIA says it tested the recipe's version combinations together on the specified platforms and hardware.

2. Choosing between two routing paths based on infrastructure ownership

One path lets Dynamo Frontend send requests directly to workers. It selects workers based on current load and the potential to reuse KV cache, which stores computation results from earlier requests. This suits application teams that own per-model endpoints or route requests through an existing proxy, a server that forwards requests. It is the simpler of the two configurations.

The other path combines Gateway API, which manages request ingress, with Dynamo EPP, which selects workers. The gateway sends requests to the selected worker, and a sidecar process forwards them to that worker. This suits platform teams managing entry paths, timeouts, retries, and fallback behavior across multiple models.

Both paths use KV cache status reported by workers. Routing requests with the same prefix to workers that already hold the computation results reduces repeated intermediate computation.

3. Moving from recipe selection to runtime checks

AICR separates runtime installation from application deployment. First, create a cluster snapshot recording its configuration, then generate an inference recipe matching the distribution, GPUs, and GPU node operating system.

aicr snapshot \
  --namespace aicr-validation \
  --output snapshot.yaml

The table below lists component versions selected by recipes for specific GPU systems and a cloud environment. The focus is on reproducing combinations tested together in that environment, rather than collecting each tool's latest version.

AICR v1.0.0 Amazon EKS recipes for NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems

Package this combination as a deployment bundle, install it, and run deployment and conformance checks. Once checks pass, install the application configuration and test the inference endpoint. Helmfile can handle installation. AICR also generates GitOps bundles to manage deployments according to repository settings.

4. Checking the foundation before app deployment

Checks vary by recipe. Dynamo checks typically cover cluster resources, GPU availability, platform component health, GPU access, scheduling, autoscaling, and monitoring metrics. Recipes using a gateway also check ingress readiness.

Offline checks compare a snapshot from another cluster against recipe requirements and skip live operational checks. Performance validation runs separately because temporary serving jobs use the GPUs being checked.

Recipes and pre-deployment checks reduce the burden of repeatedly aligning infrastructure combinations, but leave teams operating and managing pinned components together. Shared ingress enables centralized request-routing policies, but adds components to operate, including a gateway, worker selector, and sidecar processes.

AICR's core value is standardizing the runtime beneath Dynamo and checking conformance before app deployment, rather than Dynamo's inference capabilities.