NVIDIA NeMo Gym: Verification-driven instruction improvements, fewer chip design agent calls

1. Examine completed files rather than tool calls

An article NVIDIA published on October 14, 2026 describes improving the Siemens Fuse EDA AI Agent with NeMo Gym. The agent performs standard cell characterization, creating timing, power, and noise models for a chip's basic logic circuits.

Characterization results are stored in Liberty files. Even incorrect initial settings can produce plausible-looking files, so successful tool calls alone cannot establish task completion. Final verification revealed missing cells and operating conditions, incorrect signal transition time thresholds, and timing and power values outside acceptable ranges.

2. The verifier connects failure causes to the next revision

NeMo Gym is an open library that evaluates tasks, agent harnesses, verifiers, and execution state together. It starts each attempt from a clean state and logs execution, allowing comparisons of the same task under the same conditions.

The verifier in this case serves as a checklist comparing artifacts against reference data. It checks for all required cells and conditions, file usability in verification tools, failures absent from the reference data, and reuse or truncation of existing values. Only after passing these four gates does the proportion of values within tolerance become the score; failing any gate yields a reward of 0.

The team examined execution paths, failed calls, and verifier diagnostics to identify ambiguous instructions and incorrect tool choices. It then revised skills, which provide task instructions, and tool descriptions in MCP, the protocol through which the agent calls external tools. Key changes simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas.

3. Scores and efficiency improved on a limited development task

Measurements used GF180MCU, an open process design kit. The full dataset comprises 229 standard cells and 36 process, voltage, and temperature combinations, with Solido AFS used for circuit simulation.

Actual tuning runs were limited to 2 cells with the same 36 conditions. The model remained Claude Opus 4.8, and the harness remained Claude Code. On this development benchmark, the average overall verification score rose from about 55% to about 95%.

In the same evaluation, turns—the units of progress in agent conversation and tool use—fell from 155 to 53, while average output tokens dropped from about 81,000 to about 29,000. The original article reported reductions of about 66% and about 64%, respectively. These were not results from running all 229 cells, and neither measurement hardware nor monetary costs were provided.

4. Start by checking the evaluation workflow with a general-purpose example

According to the repository instructions, Python 3.13.14 or later is required. The library itself does not require a GPU, but model inference or individual task servers may need one. After installation, enter model connection details in env.yaml at the project root.

The following quickstart configuration is reproduced unchanged. It is for a general-purpose multiple-choice example, not the model configuration used in the chip design experiment.

policy_base_url: https://api.openai.com/v1
policy_api_key: <your-openai-api-key>
policy_model_name: gpt-4.1-2025-04-14

First start the local server connecting the model, agent, and scoring, then run the evaluation in another terminal. This produces execution logs and aggregate metrics. For your own design work, define valid final artifacts and build a dedicated verifier, then repeat failure analysis, description revisions, and reevaluation under the same conditions.

5. Fewer calls come at the cost of verifier design and repeated runs

Reduced output volume and fewer turns indicate improved agent efficiency. Achieving this, however, requires domain-specific verifiers that detect omissions and result reuse. The team performed about 25 runs, each taking about 1.5~2 hours. Adoption costs must account for this repeated verification work as well.

The key here is not replacing the model, but verifying errors in final artifacts and using those diagnostics to revise task instructions and tool descriptions.