llama.cpp b12122: NVFP4 gate-up fusion, CUDA path improvements

Measurement method

Prompt processing on DGX Spark improved by 28.1%. This optimization is part of b12122, released on November 5, 2026.

W4A4 uses 4-bit activations (intermediate values in model computation) and is exclusive to Blackwell. Turing~Hopper use W4A8 with 8-bit activations. In the table, master and PR mean before and after #31296. The command is shown at the end; -ub is the physical maximum batch size.

Representative measurements by GPU

Conditions: 16k, Llama-3.1-8B NVFP4, with the compute path listed per row.

Unit: tok/s

Table: Prompt processing speed by GPU

Improvements were +9.6~28.1% across 8 W4A4 combinations and +2~11.3% across 6 W4A8 combinations. The full list is in the source PR.

Prompt processing comparison with vLLM

Shared conditions: DGX Spark (GB10), nvidia/Qwen3.8-27B-NVFP4 (revision 482ca0f3), and 1 sequence.

Conditions: vllm bench latency, 1 output token, FP16 activations and KV cache (stored keys and values of prior tokens), 2048-token prefill chunks, prefix caching off, FlashInfer attention, 2 warmups and 5 measurements.

Unit: tok/s

Table: vLLM prompt processing speed

Conditions: command below, 0 output tokens, 3 measurements, F16 KV cache, NVFP4 W4A4. Attention is listed per row.

Unit: tok/s

Table: llama.cpp prompt processing speed

Why the difference?

Gate-up computes two input branches and uses the gate result to modulate the up result. MMQ, the CUDA kernel path for quantized matrix multiplication, executes both matrix multiplications and GLU, which combines their results, in one pass. Rows alternate in groups of 8 so each thread holds both up and gate values for the same output element. When writing to memory, it applies GLU and NVFP4 weight scales using registers.

Earlier work covered standard feed-forward networks in #29948, merged 2026-10-25. Expert-based MoE support followed in #31188, merged 2026-10-28. Separate DGX Spark tests showed Llama-3.1-8B Q4_K_M rising from 2749.3 → 3000.6 tok/s (+9.1%), and Qwen3-30B-A3B Q4_K_M gaining +17.2%. This work extends that approach to NVFP4.

Traces from nsys, a performance profiler, broke down the remaining gap with vLLM. The traced setup combined this PR with FP8 PR #28413 on DGX Spark using Qwen3.8-27B-NVFP4. GPU time for one 8K prompt was 5.02 seconds for llama.cpp and 2.47 seconds for vLLM. GPU utilization was 98~99% for both, so the entire difference came from kernel time.

The largest gaps were FP4 matrix multiplication (1.59 vs. 0.87 seconds), small auxiliary operations (0.96 vs. 0.26 seconds), and activation quantization before matrix multiplication (0.82 vs. 0.27 seconds). For quantization, llama.cpp reads F32, the 32-bit floating-point format, twice, taking 456ms per 8K prompt. vLLM fuses FP4 quantization into SwiGLU/RMSNorm, neural-network operations for activation and normalization. This removes the amax pass, which computes each block's maximum absolute value.

Selection criteria

The W4A4 path on Blackwell GPUs (DGX Spark, RTX 5090) showed the largest gains in the table. On DGX Spark with Llama-3.1-8B NVFP4 and a 4096-token prompt, lowering ubatch to 16 still yielded a 3.0% gain without a penalty. b12122 belongs to the continuously published b-number builds. The previous stable release, v0.9.0 (2026-11-04), supports OCP FP8 E4M3 quantization and the flash_attn_ext_rows operation. Figures are self-reported measurements by each PR author, who disclosed Claude's assistance with comparative benchmarks and tracing.

On cards supporting CUDA MMQ, compare the previous build with b12122 using the same model and settings. Replace <gguf> in the reproduction example with the model file.

llama-bench -m <gguf> -ngl 99 -fa on -p 8192,16384 -n 0 -b 2048 -ub 2048 -r 3

Operation fusion speeds up input processing, but kernel time for matrix multiplication and activation quantization still accounts for the gap.