NVIDIA released preliminary inference results for AI tasks involving repeated conversations and tool use. It reports 5–10x higher throughput per watt and 3–5x faster responses at the same concurrency than the previous generation. This suggests room for companies running agent services to expand capacity within power budgets.

1. Measuring both speed and capacity for agent workloads

MLPerf Endpoints v1.0 runs coding and enterprise workflows through APIs to assess capacity and responsiveness. Agents repeatedly reason, call tools, and respond, so a single response's speed cannot capture real task performance.

Each point in the results page's agentic report represents a run at a different user concurrency. System throughput is total tokens per second, while end-to-end interaction is the tokens per second experienced by one user across a task.

Performance runs check tool calls and workflow intent against reference values. Problem-solving ability is assessed separately with SWE-bench Verified.

2. Higher throughput efficiency in company-reported results

The comparison covers Vera Rubin NVL72 and the previous-generation GB300 NVL72. Power efficiency is throughput per 1kW of allocated system power. Each platform's best run meeting the model's minimum response speed was selected.

NVIDIA reports throughput per watt gains of 10x for Kimi K3, 9x for DeepSeek-V4.1-Flash, and 5x for Qwen3.6-35B-A3B. Per-user response speeds at the same query concurrency were reported as 3x, 5x, and 3x, respectively.

The throughput comparison requires at least 50 tokens per second per user for Kimi, 280 for DeepSeek, and 410 for Qwen. Speed gains use 1,024 concurrent queries for Kimi and DeepSeek, and 256 for Qwen. The three models also use different datasets, so each gain applies to that model and workload.

3. More concurrent queries trade responsiveness for throughput

In Kimi's preliminary submission, runs on the same 32-accelerator system showed higher throughput per allocated watt as query concurrency rose. The following three runs share the same submission and power conditions in the public data.

Table: Vera Rubin throughput per watt at different Kimi loads

Conditions: MLPerf Endpoints v1.0, Standardized division, Available category, 32 Vera Rubin NVL72 accelerators, 100.89kW allocated power, Kimi K3, agentic_combined, TensorRT-LLM and Dynamo. Query concurrency varies across online runs of the final configuration.

Units: concurrent query count; throughput per watt in tokens/second/kW

Throughput per watt at different loads on the same Kimi system512: Throughput per watt 254.6; 1024: Throughput per watt 410.5; 1536: Throughput per watt 478.6512254.61024410.51536478.6
Throughput per watt at different loads on the same Kimi system
More concurrent queries trade responsiveness for throughput
Concurrent queriesThroughput per watt
512254.6
1024410.5
1536478.6

NVIDIA reports 41,414.2 tokens per second overall and 50.4 tokens per second per user at 1,024 concurrent queries. At 1,536 queries, throughput efficiency rose, but per-user speed fell below the comparison threshold of 50 tokens per second.

The GB300 baseline for the 10x efficiency comparison used 48 accelerators, 60kW, and 48 concurrent queries. It delivered 2,210.4 tokens per second overall and 56.2 per user. The 3x response-speed comparison instead used GB300's 16.8 tokens per second per user at 1,024 concurrent queries.

4. Reusing prior conversations and reducing generation steps

NVIDIA cites fast memory access, accelerated matrix operations, and GPU communication. It also uses state reuse to avoid reprocessing conversation history, reuses prior results, and separates input processing from generation.

NVIDIA says DeepSeek's specialized compute programs reduce data movement. For Qwen, proposing several tokens and verifying them together reduces sequential generation steps.

5. Connecting efficiency gains to power budgets

The benefit is room to serve more agents within a power budget while meeting the same response-speed requirement. The trade-off is total system power. In the Kimi comparison, Vera Rubin's allocated power is 100.89kW versus GB300's 60kW. Deployment reviews must consider facility power capacity alongside efficiency gains.

MLCommons Association has validated the GB300 results. Vera Rubin's figures are preliminary, await peer review by experts, and may change.

The key is to set the service's required response speed first, then choose the capacity per watt that meets it, rather than focus on maximum throughput.