llama.cpp Gemma 4: Prefill optimization, improved throughput

1. Less work in layers that reuse earlier results

Merged on November 3, the change skips token computations not needed for output in layers that reuse the KV cache of earlier results without storing new values. During prefill, which processes input tokens in advance, these layers are skipped for intermediate batches, or chunks of input. Only one row is processed in the final batch, reducing redundant work where existing results are reused.

The developers also streamlined scaling for per-layer embeddings, the input representations supplied to each layer, and avoided contiguous data copies.

The code below shows the conditions used in the original change. It checks for cache-sharing layers and a single input sequence, among other conditions, to avoid pruning necessary intermediate results.

    // KV-shared layers do not write to the KV cache, so prune non-output token rows before the first KV-shared layer
    bool prune_shared = crop_before_nextn(inp_out_ids) &&
                        hparams.n_layer_kv_from_start > 0 && hparams.n_layer_kv_from_start < n_layer &&
                        !cparams.embeddings && cparams.n_seq_max == 1;
    for (int il = (int) hparams.n_layer_kv_from_start; prune_shared && il <= n_layer; ++il) {
        prune_shared = !cparams.embeddings_layer_inp[il];
    }

2. E2B long-input throughput rose on M4 Pro

The PR author compared the pre-change master development baseline with the modified version using official Google QAT Q4_0 models, trained to account for quantization effects. E2B tests on M4 Pro with Metal and BLAS and 10 threads showed large prefill throughput gains, measured in input tokens per second, at all three input lengths.

Table: E2B prefill throughput by input length

Conditions: M4 Pro, MTL,BLAS, 10 threads, gemma4 E2B Q4_0, 3.10 GiB, 4.63 B

Unit: tokens per second; ± columns show the error component of the original figures

In the same setup, 128-token generation stayed similar at 109.65 versus 110.98 tokens/s. The main improvement is in reading input rather than continuing an answer.

3. E4B also improved across input lengths

The author's M4 Pro measurements showed the same trend for E4B. The table below covers E4B only, with the same hardware and thread settings as the earlier test but a different model size and parameter count.

Table: E4B prefill throughput by input length

Conditions: M4 Pro, MTL,BLAS, 10 threads, gemma4 E4B Q4_0, 4.79 GiB, 7.46 B

Unit: tokens per second; ± columns show the error component of the original figures

In the same setup, 128-token generation throughput was 63.39 versus 63.62 tokens/s. E4B likewise combined higher input throughput with steady generation speed.

The author reported unchanged outputs for text, image, and audio inputs, and also tested prompts of 16K and 32K in length.

4. Input throughput also rose on CPU and Snapdragon

The author's M4 Pro CPU test used the same official quantized models and 10 threads. For 512-token inputs, E2B rose from 629.04 to 1811.26 tokens/s, and E4B from 326.06 to 543.23 tokens/s.

The Snapdragon S26+ test used OpenCL and HTP, 6 threads, a microbatch size of 1024, the HTP0 device, and ngl set to -1 to control the number of layers offloaded to the GPU. With the same official E2B model, throughput for 512-token inputs rose from 2303.07 to 5099.01 tokens/s, and for 8192-token inputs from 1896.88 to 4686.80 tokens/s. Each platform's results show before-and-after differences within its own environment.

5. Benefits depend on the server's concurrency configuration

In M4 Pro server tests with Metal published by the author on October 30, processing a 13,789-token input took 4.22 seconds with unified KV, a combined cache, versus 11.72 seconds with split KV, a divided cache. With multi-token prediction (MTP) disabled and enabled, respectively, throughput was 3261.9 and 3264.4 tokens/s for unified KV, versus 1175.2 and 1175.0 tokens/s for split KV.

At merge time, pruning requires unified KV and a single sequence, meaning one input stream at a time. Configurations using multiple sequences, as the default server does, or extracting embeddings are excluded. The author explained that 26B does not benefit because all its layers write values to the cache during prompt processing.

The trade-off for reduced input computation is a narrower range of supported deployment configurations. Servers handling concurrent requests must weigh existing concurrency needs alongside single-sequence input throughput gains.

This optimization reduces long-input latency rather than answer-generation time, and adoption should be judged by the model and server configuration.