llama.cpp: Fused QSA selection, kq_mask allocation removed

1. Removing an unnecessary buffer from model attention processing

Pull request 30965, a request to review code changes, was merged on October 23, 2026, and applies to qwen4exp. The submitter explained that all 12 attention layers use QSA, so a separate kq_mask allocation is unnecessary.

Instead of allocating space for separate markers, the approach fuses the required selection results. The commit description also reports a fix for a memory reservation error caused by mismatched selection array shapes when using multiple processing streams.

2. Memory reductions in compute buffers for temporary results

Submitter sanmai compared the existing master build, an executable built from the baseline code, with the modified build, counting only compute buffers. In the table below, ubatch is the number of tokens processed at once.

Measurements were taken on a 5090@400W. CUDA0 denotes the GPU-side buffer, and CUDA_Host denotes the host-side buffer.

Table: Compute buffer memory by build

Conditions: Qwen3.8-Flash-Next UD-Q4_K_XL, 5090@400W, --load-mode dio, -cmoe --moe-cache-mib 8000 -ctk q8_0 -ctv q5_1

Unit: MiB

Both buffers were smaller in the modified build. The submitter summarized the savings as 2GB each for GPU and host memory at a token batch size of 4096, and 4GB each at 8192. Temporary workspace for model computation shrank on both sides.

3. Speed and real-model validation

The submitter reported a 40% increase in token generation speed with other conditions unchanged. The published modified-build run used ubatch 4096 and an expert cache, temporary storage for model expert information, of 17750 MiB. Input processing reached 1004 tokens per second, generation reached 45.5 tokens per second, and peak graphics memory usage was 32066 MiB.

With ubatch 8192 and an expert cache of 13400 MiB, input processing reached 1470 tokens per second, generation reached 41.2 tokens per second, and peak graphics memory usage was 32036 MiB. Both runs used the previously stated 5090@400W and the same loading method.

The original speed table provides no baseline figures for the existing build, and the two runs use different cache settings, making direct comparison difficult. The compute buffer table also measures temporary compute space, not total memory.

Approving reviewer ServeurpersoCom reported testing with an actual UD-Q4_K_XL model. At a context size of 16384, the input range the model consults, logits—the scores for next-token candidates—were byte-for-byte identical to the existing build. At a context size of 262144 and a token batch size of 8192, the compute buffer and pinned host buffer each shrank by 4GiB.

4. Checking conditions with the published command

The submitter published the command used for the Qwen3.8-Flash-Next UD-Q4_K_XL model. It corresponds to the measurement with an expert cache of 13400 MiB and also shows the loading method and token batch size.

llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
    --no-mmproj -ctk q8_0 -ctv q5_1 --cache-ram 32768 \
    -b 8192 -ub 8192 \
    -cmoe --moe-cache-mib 13400 --load-mode dio

The submitter expected users of 16GB and 24GB graphics cards to notice the greatest benefit. In this view, the value of the freed space depends on how close the workload runs to the memory limit.

5. What model-specific optimization means

The benefit is reduced GPU and host compute space, while the required work is developing and validating model-specific execution paths. This change included array shape fixes and logit comparisons using a real model. The optimization focuses work on implementation rather than combinations of user settings.

The change passed 19 automated checks and was merged into master. Developers using the affected model can now benefit from the memory savings through a build that includes the merge.

This merge removes buffers that QSA models do not need, creating practical headroom, especially for workloads running close to memory limits.