NVIDIA shared a process in which AI agents help design obstacle courses and prepare 3D assets, while robots learn and validate walking in virtual environments. Developers can test courses and robot movements before building physical environments. Training and validation help prepare for transfer to real robots.

1. From prompt to course plan

NVIDIA introduced a six-step workflow using Microduck as an example. Plans are prepared on RTX Spark, the local development hub, and validation returns to the local system after training.

Hermes and NVIDIA Nemotron 3.5 Bolt turn a prompt into a plan covering tasks, resources, deliverables, and execution order. They also define the course layout, obstacle dimensions, passage widths, and routes.

The original article provided a sample prompt for starting the work.

Build and test a simulated Microduck obstacle course with navigation and locomotion challenges. Make a photorealistic concept image and have Astra build and test it.
...
Workflow
Use Hermes and Nemotron 3.5 Bolt for preparation and Astra for asset creation and simulation.

Planning includes reviewing robot documentation and repositories to identify capabilities, relevant resources, and issues needing further checks. Assets to reuse are then selected from a course concept image, with cropped or obscured areas recorded. Hermes bundles the plan, image, asset list, and references for handoff to Astra.

2. Agents create assets and assemble the simulation

Astra uses cloud image editing to clarify low-resolution references cropped from the concept image, which have limited detail. It compares them with the original to check that silhouettes, or outer outlines, proportions, colors, and features remain intact. Dimensions and hidden surfaces not visible in the image remain modeling assumptions: estimates for details missing from the references.

FreeCAD creates the basic shapes and editable structure, while Blender adds detailed geometry and materials. Both programs run locally and programmatically, automating tasks through code. Outputs are saved as editable models and simulation assets, then previewed and inspected.

Existing assets are another option. According to the company, the free SimReady Central Library contains more than 2,000 simulation-ready 3D assets.

Course assembly uses Isaac Sim's rendering, which turns scene information into images, and sensor features, alongside the Newton physics engine for motion and collisions. Once the relationships among the robot's body, joints, and actuators are configured, the simulation calculates movement and balance. Joints connect body parts and allow rotation, while actuators turn power into motion. Agents can help with this configuration.

3. Training a walking policy in parallel environments

A policy is a set of rules that turns a robot's observations into actions. Isaac Lab runs multiple simulations concurrently in the cloud to gather experience for improving a single policy. Developers implement tasks by defining what the robot observes, which actions it can take, and how to reward successful behavior with scores that encourage the desired actions. Falls and collisions in simulation also inform policy updates, which adjust behavioral rules based on experience.

This example started with a pretrained locomotion policy from Isaac Lab's Microduck example environment. Further training taught it to traverse curved ramps, structures with curved slopes, and obstacles. Observations included depth information from a head-mounted camera, indicating distances between the camera and objects, along with robot state information.

According to the company, this additional training used 4,096 parallel environments on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU. Attempts by multiple robots were gathered simultaneously to update the policy.

4. Validating locally and preparing for real hardware

The trained policy is evaluated in Isaac Sim on the local RTX Spark. Evaluation checks whether the robot detects obstacles, adjusts its movements, and completes the course, recording falls, collisions, and stalls. Reviewing execution videos alongside logs can reveal where to improve the environment or training settings.

Before deployment on a real robot, the simulation can be adjusted to better reflect real-world behavior. Isaac Lab's Better Actuator Models model command latency, the time from command delivery to actuator response, backlash, and actuator responses to motor voltage drops. Backlash is the play caused by small actuator gaps when movement reverses direction. Settings can be matched to real measurements, with domain randomization varying physical conditions and observation noise within realistic ranges. This can train the policy to handle differences between simulation and reality.

NVIDIA also showed the trained walking policy running on a real Microduck. Differences found in real-robot tests feed into subsequent simulation and training.

The original article does not provide the success rate or training time for this additional training run. Detailed procedures for deployment on real hardware are also outside its scope.

5. Reducing preparation work and strengthening validation

Agents can help plan courses and create assets, while ready-made assets can further reduce the burden of building environments. Human review must therefore focus on modeling assumptions not supported by images, asset suitability, and simulation results. Since cloud training connects local preparation and validation, developers must plan both asset inspection and training environment operations.

NVIDIA's approach automates preparation for robot training while making validation of the gap between simulation and real hardware part of development.