Key-value caches store intermediate information reused during generation. llama.cpp added a GPU compute path that uses compressed key-value caches to generate responses without fully decompressing them. Developers running local language models on supported GPUs can reduce cache conversions repeated during generation.
1. Read the stored cache directly instead of decompressing it
Released on October 29, 2026, b11980 centers on a memory-saving flash attention kernel added to the SYCL backend. SYCL is a parallel computing standard for heterogeneous processors.
Previously, quantized caches were unpacked element by element or converted entirely to 16-bit floating point on every call. The new path reads data as needed in its stored form rather than unpacking everything. Supported key-value cache formats are q4_0 and q8_0.
Core operations use the DPAS matrix computing capability. Query-key products use 8-bit and 4-bit integer combinations or pairs of 8-bit integers. Products of probabilities and values, the information used in attention's weighted sum, remain in 16-bit floating point, a format that represents fractional numbers.
2. Checking model configuration is the starting point for adopting the generation path
The release notes describe support for head sizes of 64~512, the vector width of an attention head. They also list grouped-query attention up to 32, where multiple query heads share keys and values. Attention sinks, a technique that anchors attention to initial tokens, are supported too. However, a subsequent change entry also lists the removal of head size 192 support. The new path targets single-token generation and small batches with 1~31 query tokens. Prompt processing retains existing kernels.
3. Throughput increased under the same generation conditions
The project's release notes presented generation performance measured on a B580 using the Llama 3.1 8B model.
Model quantization was Q4_K_M, and the key-value cache format was q4_0. Context was set to 16K, with 128 generated tokens. Under these conditions, the project reported throughput rising from 31.3 to 54.2 tokens per second.
Table: Generation throughput before and after the new path
Conditions: B580, Llama 3.1 8B Q4_K_M, q4_0 key-value cache, 16K context, 128 generated tokens
Unit: tokens/second
| State | Generation throughput |
|---|---|
| Before | 31.3 |
| After | 54.2 |
Chart: Generation throughput comparison (same conditions and unit as the table above)
Before | ████████████████ 31.3
After | ███████████████████████████ 54.2Calculated from the original figures, throughput after the change was about 1.73x the previous level, an increase of about 73.2%. More tokens were generated per second under the same conditions. Developers testing their own environments can use this as a reference for matching hardware, model quantization, cache format, context length, and generation length.
Test results were also included. Of 4584 FLASH_ATTN_EXT tests, 4582 passed. The release notes said the 2 failures also occurred on the existing development branch. These results, presented alongside throughput, provide a reference for reviewing the new path's computational behavior.
4. Avoiding repeated conversions translates into generation performance
The new path operates directly on stored blocks, avoiding the full-cache conversion performed on every call by the previous path. The project's reported generation throughput gains offer evidence for examining this conversion-avoidance design.
The distribution list separates Ubuntu x64 SYCL packages into 32-bit and 16-bit floating-point versions and also includes Windows x64 packages. Developers already using this backend can explore the new path starting with those packages and their own cache configurations.
The release notes also provide a setting to turn off the new path. Setting the following environment variable value disables it, preserving a choice when examining generation behavior after adoption.
GGML_SYCL_FA_DPAS=0The value of b11980 lies in providing a SYCL compute path that reduces repeated conversions in generation workloads using quantized caches and translates that into actual throughput gains.