NVIDIA's tutorial, released on October 30, 2026, shows how to launch a local OpenAI-compatible API server with a few commands to test agents on-device instead of using cloud APIs. In the company's tests, enabling acceleration that proposes candidate tokens in advance barely changed initial response latency but sped up subsequent text generation.
The model tested was Nemotron-3.5 Lightning running on DGX Spark 64GB. All figures below are as reported by NVIDIA.
Results
Conditions: DGX Spark 64GB; DGX OS 7.6.0 (Ubuntu 24.04.4 LTS); Edge-LLM 0.11.1; Nemotron-3.5 Lightning 30B-A3B NVFP4 (4-bit floating-point quantization). Reasoning via thinking enabled; batch size 1; 1 concurrent stream; output capped at 128 tokens; 3 candidate tokens with DSpark. Average inputs in table order: 1,153/2,247/139 tokens. Source: NVIDIA technical blog, Table 1.
Table: Generation speed and time to first token with and without DSpark
Units: Generation speed in tok/s, time to first token in ms
| Decoding workload | Mean generation speed without DSpark | Mean generation speed with DSpark | Median time to first token without DSpark | Median time to first token with DSpark |
|---|---|---|---|---|
| SpeedBench 1K | 73.3 | 108.5 | 201.0 | 203.0 |
| SpeedBench 2K | 72.8 | 105.3 | 329.5 | 331.6 |
| HumanEval prompts | 73.4 | 126.9 | 98.1 | 98.7 |
Calculated from the table, HumanEval generation speed rose from 73.4→126.9 tok/s, about 72.9%, while the two SpeedBench results rose by about 44.6~48.0%. Median time to first token (TTFT) differed by no more than 2.1ms in all cases. Generation speed excludes time to first token, so the gains centered on subsequent token generation rather than the initial response.
TTFT was longest on SpeedBench 2K, which had the longest input, and shortest on HumanEval. This order held with DSpark enabled, so the added cost of verifying candidate tokens did not substantially increase first-token latency.
Performance documentation comparison by device and method
Conditions: TensorRT Edge-LLM 0.10.0 runtime throughput dashboard; llm_inference measurements; Nemotron-3.5-Lightning-30B-A3B NVFP4; batch size 1; input processing length 66 tokens. Source: NVIDIA's official performance documentation.
Table: Generation speed and average acceptance length by inference acceleration method and device
Units: Generation speed in tok/s, average acceptance length in tokens/verification step
| Inference acceleration method | DGX Spark (GB10) generation speed | Average acceptance length | Jetson AGX Thor generation speed | Average acceptance length |
|---|---|---|---|---|
| Baseline generation | 75.6 | — | 81.6 | — |
| MTP | 93.2 | 2.70 | 95.3 | 2.69 |
| DFlash | 61.5 | 2.23 | 58.7 | 2.22 |
MTP trains a model to predict multiple upcoming tokens, while DFlash uses a draft model to generate entire candidate blocks. Average acceptance length is the number of tokens accepted per verification step. MTP was faster than baseline generation on both devices, but DFlash was slower. Speed gains depend on candidate generation and verification costs as well as accepted token counts, so acceptance length and final generation speed should be considered together.
Why the results differ
Enabling DSpark reduces the number of repeated runs of the main model. A draft model first proposes a block of candidate tokens, and the main model verifies that block in one pass.
Selection criteria
For interactive requests, the Nemotron-3.5 Lightning model card recommends DSpark as the default for DGX Spark and low-concurrency, latency-sensitive serving. Depending on the task, disabling acceleration or using DFlash or MTP may work better, so comparing different candidate token counts is advisable. For data center batch throughput, NVIDIA's Nemotron-3.5 Lightning model card says vLLM with the humming backend is fastest without speculative decoding in a maximum-throughput batch on 1× H100. This recommendation applies to H100 batches.
Running locally
The tutorial assumes CUDA 13, TensorRT 10.16.1.11, and Python 3.10~3.12. Install tensorrt-edgellm[server]==0.11.1 in a virtual environment with access to system packages, then start a conversation with this command.
trtedge chat nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4Launch the local API with trtedge serve using the same model ID. Use the same request to compare a baseline with acceleration disabled against results with 3 candidate tokens enabled.
Faster generation and the wait for an initial response are separate concerns, so local chat acceleration is best assessed using both metrics separately.