A new llama.cpp preview build speeds up generation by moving more Zamba2-family computations onto the GPU. It targets developers running these models with CUDA. Developer measurements showed about 35% more throughput on the same hardware and with the same quantization settings. Quantization stores model values at reduced precision, while throughput measures work processed over time.

1. The GPU now handles previously unsupported computations directly

The key change closes a gap between the model's computations and GPU support. Zamba2-like architectures use a state size of 64 in recurrent scan steps that carry forward earlier state. State size is the number of values retained during computation. The existing CUDA path supported only 96, 128, and 256, so the model's main scan computations fell back to the CPU at every step.

The release description says Zamba2 processed 6~7 tokens per second under this bottleneck. Other architectures, or model structures, delivered 50~70 tokens per second on the same hardware. This illustrates how CPU processing can limit generation speed even with a GPU if the required computation path is missing.

The fix follows the existing implementation pattern to add a path for state size 64. It also updates the GPU support checks and error message. The mathematical computations remain unchanged.

2. Inference measurements under matching conditions showed about 35% higher throughput

The PR author reported that Zamba2-7B-Instruct throughput rose from about 11 to 15.1 tokens per second. Inference, the process of generating output from input, showed a reproducible gain of about 35% with hardware and quantization unchanged. The release notes and commit description, which records the code change, include the same before-and-after figures.

The result shows the gain after removing a major cause of CPU fallback. Rather than changing the model's mathematical operations, this fix raises throughput by expanding the computations the GPU can handle.

3. Accuracy was checked against CPU reference results

The PR author said GPU results were compared with the CPU reference implementation in the built-in correctness tests. Three new tests reflect Zamba2's actual dimensions, the sizes defining its computation structure, and all 28 tests passed. This checks that results match the reference calculations as GPU coverage expands.

The added tests cover both multi-token processing and single-token generation. The following excerpt from the original code selects the GPU computation for state size 64.

        if (d_state == 64) { // Zamba2
            constexpr int threads   = 64;
            constexpr int num_warps = threads/WARP_SIZE;

            const dim3 blocks((n_head * head_dim + (num_warps - 1)) / num_warps, n_seq, 1);
            const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params(blocks, threads, 0, stream);
            ggml_cuda_kernel_launch(ssm_scan_f32_group<64/WARP_SIZE, 64>, launch_params,
                    src0, src1, src2, src3, src4, src5, src6, dst,

4. CPU overhead remains despite reducing the main bottleneck

The benefit of this change is that the GPU directly handles the main scan computations. The optimization changes which device performs the work by adding a previously CPU-bound path to GPU support.

However, the PR author reported that about 28% of scan calls still fell back to the CPU for a separate reason, leaving throughput below that of other architectures. The source did not explain the gap between 6~7 tokens per second during bottleneck investigation and about 11 in the improvement measurements. It also did not specify the GPU or quantization format.

Remaining CPU processing is a cost to consider alongside the gains. Even after moving the main computation path onto the GPU, some work remains on the CPU. Further optimization therefore focuses on identifying the separate causes of the remaining fallback.

5. Available in CUDA builds of the preview release

b12207 is a preview release published on November 8, 2026. Its release assets include CUDA builds for Linux and Windows. Developers running Zamba2 in these environments can select the CUDA build for their operating system.

The key consideration when choosing a build is that this fix affects the CUDA computation path. The release offers CUDA 12 and 13 builds for Linux and Windows, along with links to the accompanying CUDA libraries.

The value of b12207 lies in improving Zamba2 generation throughput by closing a GPU support gap without changing the mathematical computations.