NVIDIA has integrated an AI workload inventory collector into its GPU platform configuration tool for Kubernetes. Platform teams can manage reproducible deployments and detected model and runtime data in the same deployment flow.
1. Connecting configuration management and runtime inventories
NVIDIA AI Cluster Runtime (AICR) is an open-source module of the DSX AI Factory platform. Given a cloud, accelerator, OS, and intent, it generates a version-pinned deployment recipe and installation files. The recipe can be checked against actual cluster state.
The integration adds Google's open-source k8s-aibom to AICR's validated components. AICR pins source code, installation packages, controllers, and cluster automation component images to specific versions and contents for testing. It checks build provenance and permissions, then tests installation, upgrades, and removal.
The tools examine different things. AICR checks that GPU infrastructure uses a known configuration; k8s-aibom collects assets from recognized AI workloads. This places deployment intent alongside discovered workload data. The same image tag can point to different code across nodes or download times. A pinned infrastructure recipe alone therefore cannot track workload changes.
2. Recording detected AI assets and evidence
k8s-aibom observes Kubernetes workloads and their execution units, pods. For each recognized AI workload, it creates an AI asset inventory in CycloneDX 1.6, a format for exchanging software composition information.
Support covers inference services, agents, retrieval-augmented generation setups that use search results to generate answers, vector databases that retrieve by semantic similarity, training jobs, and evaluation tools.
When a child workload is created by a parent job, its inventory is consolidated into the parent's document to reduce duplicate records.
Inventories contain observed properties and evidence, including models, container images, and runtimes. Names in model runtime arguments are marked as declared; information read from images or conventional environment variables is marked as inferred.
Unresolved values are marked separately. Signature verification checks provenance and changes using signatures. Model information can receive a verified grade if it passes the team's trust and signer requirements. NVIDIA disables this verification by default.
3. Selecting it in recipes and labeling namespaces
The AICR v1.2.0 catalog described in the blog pins k8s-aibom 1.6.2. Use a matching command-line tool when reproducing the example.
The example uses a Google Kubernetes Engine (GKE) cluster running Kubernetes 1.32 or later. Two H100 inference recipes with Container-Optimized OS (COS) select the collector by default. Other GKE recipes can select it at generation time. The recipe command is:
aicr recipe \
--service gke \
--accelerator h100 \
--os cos \
--intent inference \
--output recipe.yamlAfter installation, label namespaces, which partition resources within the cluster. Inventories are generated for recognized, supported workloads in labeled namespaces.
4. Collection overhead and operational considerations
According to the company, the upstream project measured k8s-aibom v1.6.1 in a GKE setup with NVIDIA Dynamo and AICR. The test used CPU-only nodes with workload pods remaining Pending, so these figures describe controller inventory overhead without any workload pod running successfully. Steady state was observed for 15 minutes.
Table: Controller resource usage at steady state
Conditions: k8s-aibom v1.6.1, GKE, NVIDIA Dynamo and AICR setup, CPU-only nodes, workload pods remaining Pending (not running), 15-minute steady-state observation
Units: API server requests per minute, CPU in millicores, memory in MiB
| Metric | Value |
|---|---|
| API server requests (approx.) | 4 |
| Average controller CPU | 2.2 |
| Controller working-set memory | 28–34 |
The figures show inventory controller overhead, not AI model throughput. The chart's default resource requests, a separate allocation baseline, are CPU 50 millicores and memory 128MiB, above the observed load. These defaults are not measurements or confirmed settings for the v1.6.1 test. In that test, changing a workload's image tag regenerated the inventory once.
The controller retains pod specifications and directly configured environment variable values in memory cluster-wide, including unlabeled namespaces. External storage transfers and access to secrets are not configured by default.
5. Linking deployment intent and observations
Inventories cover only supported workloads with recognizable AI signals and may appear ready before pods download images. They list declarations and discoverable information, rather than confirming scheduling or successful execution.
Comparing AICR recipe intent with observed runtime assets requires shared asset identifiers, freshness criteria, and aligned infrastructure and workload scopes. NVIDIA presents this connection as a future integration task.
This integration adds AI asset evidence to reproducible GPU deployments, giving operations teams a starting point for reviewing infrastructure and workloads together.