NVIDIA shared three optimizations to speed up language model response generation on DGX Spark. The company reported 42.5% higher generation throughput and 24.5% lower average latency on a full question set. These results offer developers a benchmark for tuning local inference environments.
1. Reduce the read burden of expert computation and the output layer
The model evaluated was Inclusion AI's Ling-3.0-flash-fp4. This MoE model computes with only selected experts, activating 5.1 billion of its 124 billion parameters per token. Analysis on DGX Spark at batch size 1, processing one request at a time, found that token generation was limited by weight-reading speed.
The main costs were expert computation and the output layer that scores vocabulary entries. NVIDIA replaced the kernels, programs that execute operations on the GPU, and compressed output-layer weights to reduce reads per step. It also added speculative decoding, which proposes multiple tokens and verifies them together, to reduce generation steps.
2. Combine kernel replacement with an FP8 output layer
The first method replaces the default FlashInfer MXFP4 expert computation path, which uses a blockwise 4-bit floating-point format, with Humming kernels. Humming compiles kernels at runtime for the matrix size, data format, and GPU, then caches them for reuse. The company says individual kernels compile in seconds, while the default path takes tens of minutes to compile 97 targets.
The second method converts only the output layer from BF16, a 16-bit floating-point representation in the saved model, to the smaller 8-bit FP8 format during setup. For every generated token, the output layer reads all weights corresponding to 157,184 vocabulary entries. FP8 halves this layer's read volume. Quantization settings for the rest of the model remain unchanged.
The baseline used 4-bit weights and 8-bit activations with a path that outputs scores in FP32. The FP8 output layer replaces this path.
This combination reduces per-step costs and generation steps, but adds compilation and caching for new kernels and draft verification work. Adoption decisions should weigh token generation gains against these preparation and verification costs.
3. A built-in prediction layer drafts tokens for the main model to check
Ling-3.0-flash's NEXTN prediction layer predicts next-token candidates and can generate drafts without additional training or downloads. When the draft proposes three tokens in sequence, the main model checks four positions together and accepts the matching prefix. This confirms multiple tokens in one check.
The company reported that one verification cost about 2.26x as much as one normal generation step. At batch size 1, expert activation was a greater burden than reading cached information from previous tokens. Normal generation uses 8 experts per token, while verifying four positions activates up to 32.
Combining the FP8 output layer with speculative decoding requires the draft model to share the main model's entire output-layer module, not just its weights. Without this sharing, FP8 weights are interpreted as BF16, reducing the accepted length to 1.00 tokens and potentially making inference slower than before optimization.
4. Synthetic tests showed large cumulative gains at each stage
NVIDIA added each method in sequence to a synthetic workload with 32K input tokens and 2K output tokens on DGX Spark. The company reported throughput of 102.88 tokens per second with all three methods, up 185.1% over the baseline.
Table: Cumulative optimization results for the synthetic workload
Conditions: NVIDIA DGX Spark, Ling-3.0-flash-fp4, 32K input and 2K output tokens, batch size 1, greedy decoding, speculative configuration steps=3 / topk=1 / draft_tokens=4
Units: time per token in ms, throughput in tokens/second, increase in %
| Configuration | Time per token | Throughput | Increase over baseline |
|---|---|---|---|
| Baseline | 27.71 | 36.09 | — |
| Add Humming | 25.35 | 39.45 | +9.3 |
| Add FP8 output layer | 22.76 | 43.94 | +21.7 |
| Add speculative decoding | 9.72 | 102.88 | +185.1 |
This test selected the most likely tokens, producing repetitive output. Speculative decoding accepted an average of 3.90 tokens at a time, confirming multiple tokens together.
5. Throughput rose 42.5% on the full question-set evaluation
The full GSM8K evaluation covered 1,319 questions with 5 examples and reasoning generation. Sampling, answer extraction, random seeds, and memory allocation were matched across both configurations. The company reported average output lengths of 567 tokens for the baseline and 607 after optimization.
Table: Baseline and optimized configurations on the full GSM8K evaluation
Conditions: NVIDIA DGX Spark, Ling-3.0-flash-fp4, 1,319 GSM8K questions, batch size 1, 5 examples with reasoning generation. Temperature 0.6; cumulative probability threshold 0.95; top 29 candidate tokens by probability; maximum output 32,768 tokens.
| Metric | Baseline | All three methods |
|---|---|---|
| Time per token (ms) | 26.45 | 18.56 |
| Decoding throughput (tokens/second) | 37.81 | 53.89 |
| Accuracy (%) | 96.89 | 96.82 |
| Average latency (seconds) | 15.21 | 11.49 |
| Average time to first token (seconds) | 0.207 | 0.214 |
The company reported no statistically detectable change in accuracy. This evaluation sampled tokens by probability, with a median output length of about 310 tokens, shorter than in the synthetic test. An average of 2.68 tokens were accepted at a time, explaining the difference in gains between the two tests.
The key adoption criterion is to verify the combined effect of all three optimizations using inputs and outputs resembling deployment, rather than the peak gains in synthetic tests.