llama.cpp PR #30715: IQ*_K Quantization, CPU Inference

1. What the proposal adds

The proposal adds five formats, from IQ2_K to IQ6_K. Quantization compresses weights by representing them with a small set of values. The new formats use nonlinear codebooks, tables whose representative values are unevenly spaced.

Each block of 256 weights is split into groups of 16, storing per-group scales and information for shifting the codebook by fixed intervals. The codebook has 2 to the power of n levels, where n is the target bit count, so 4 bits means 16 levels.

The original designer first developed these formats in ik_llama.cpp. This proposal follows their block layout and names, but its codebooks and quantizer implementation were developed separately.

2. Usage and compatibility

The test repository provides a conversion command. The command below specifies calibration data to tune weight compression. Replace <TYPE> with the name of the selected quantization format.

llama-quantize --pure --token-embedding-type q8_0 --output-tensor-type q8_0 --imatrix imatrix/imatrix-K2-Horizon-7B-medium.gguf K2-Horizon-7B-F16.gguf K2-Horizon-7B-<TYPE>.gguf <TYPE>

Sharing names with existing implementations does not mean the files are compatible. The test model notes explicitly state that these files cannot be used with standard llama.cpp. The proposer said using the existing codebooks under the same names could enable compatibility. Keeping the new codebooks would instead warrant renaming the formats. The format ID order must also match, they added. No final choice was presented.

3. Quality results and trade-offs

Quality comparisons used a single dense model, K2-Horizon-7B, with 9 billion parameters. The recipe, calibration data, and quantization settings were fixed. Evaluation used 275 chunks of 1024 tokens. Mean KLD in the following table measures the difference between the original and compressed models' output probability distributions; lower is better.

Table: Output error at equal file sizes

Conditions: K2-Horizon-7B 9B dense model; fixed recipe, calibration data, and quantization settings; KLD evaluation on 275 chunks of 1024 tokens; one test machine (hardware specifications not provided)

Units: File size in GiB; mean KLD is dimensionless

In this test, all three pairs had equal file sizes, but the new formats had lower mean error. In particular, mean KLD for the 3-bit pair fell from 0.179783 to 0.087475. These results cover only one model and one evaluation setup. Further testing is needed to see whether the improvements hold for other architectures.

Size and quality do not necessarily improve together. In the same model test, IQ6_K was 51.75MiB, or 0.69%, larger than Q6_K, while mean KLD was 55.7% lower.

By contrast, IQ2_K was 4.9% smaller than Q2_K, but mean KLD was 18.5% higher. At the lowest bit count, reducing size came at the cost of increased error.

Conversion time is another cost. Measured wall-clock time with 16 threads was 155.3 seconds for IQ3_K and 15.6 seconds for Q3_K. This is a one-time cost per model, but reducing error at the same size can substantially increase preparation time.

4. Current scope of the proposal

The change is a public pull request that has not yet been merged. Proposals to add accelerator support are separate from the CPU changes.

Published throughput tests ran the same model using the Metal backend with 12 threads. Only the 3-bit format was clearly faster; the others were generally slower or had mixed results. This is not evidence of improved CPU speed, and no CPU inference performance figures were provided.

This is an experimental proposal whose error reductions at equal sizes must be weighed against the additional conversion cost.