llama.cpp b12180: Kimi-K3.1-Pro support, DSA indexer cache

1. Input selection builds on the existing architecture

Release b12180 supports the Kimi-K3.1-Pro architecture by reusing K3's computation graph, which defines connections between operations, and its converter. It replaces attention layers that reference all inputs and calculate their relationships with DSA, where an indexer first selects a subset of keys to attend to. The indexer acts as a search component that narrows down which inputs to focus on.

In this text checkpoint, a file storing the model's trained state, only some layers include an indexer, while the other existing attention layers remain unchanged. The release specifies the following configuration.

Table: Indexer configuration for the Kimi-K3.1-Pro text checkpoint

Conditions: Kimi-K3.1-Pro text checkpoint

This configuration combines existing layers with indexer-equipped layers rather than replacing the entire model with the new approach. The table lists structural settings for the same checkpoint, distinct from the validation results discussed later.

2. Reusing the computation path adds cache requirements

Instead of gathering selected keys separately, the implementation applies a mask across the entire cache to mark entries excluded from computation. This reuses the existing sparse flash attention path, which computes quickly over a limited set of entries. Memory holds a key-only cache for DSA layers and an indexer key cache, alongside the state that existing layers carry forward from earlier computations.

Developers gain a way to connect indexer selections to the existing sparse computation path. In exchange, they must manage additional per-layer indexer tensors and key caches. Outside the sparse path, masking still leaves the cost of comparing all cached keys with queries, the search vectors the model produces for inputs.

When adopting this change, it is important to check both model support and whether execution actually uses the sparse path. Resource allocation requires considering both the reduced selection size and the scope of computation performed at runtime.

3. Conversion and quantization also account for selection accuracy

The converter reads the checkpoint's indexer configuration and maps it to internal settings. It also verifies that the list of layers receiving indexers exactly matches the existing full-attention layers, checking correspondence during model conversion.

Three types of indexer tensors used in selection are excluded from quantization overrides, which change settings for reducing numerical precision. The release cites the sensitivity of top-key selection to these tensors' precision. This accounts for selection accuracy, unlike applying general quantization settings uniformly to the indexer as well.

Expert weights, the learned values in modules assigned specific computations, are repacked while retaining the MXFP4 quantization format, as in K3. They are not dequantized, a process that restores compressed numerical representations to their original precision. Indexer tensors and expert weights therefore receive different treatment within the model.

4. Validation focused on conversion consistency and selection behavior

According to the release notes, comparison of 8 layers extracted from the Hugging Face checkpoint found 38 tensors to be exact matches. Indexer metadata, which describes settings, was also checked against the configuration. Architecture test fixtures and the full architecture test suite passed as well.

Selection results were compared against full attention, which references every input position. Perplexity (PPL), a measure of language prediction error, matched when the selection count exceeded the context length, but results changed when it was forced below that length. This validates both the converted weights and behavior when the selection range changes.

The converter skips the vision tower, which processes image features, and visual connection components, so this support is limited to text. The release provides no measured throughput, memory usage, or specific PPL values.

b12180 connects the new model's input selection structure to existing execution infrastructure, so adoption should weigh model compatibility, sparse computation, and cache costs.