NVIDIA released sparse attention design guidance for developers of models that process long documents and large codebases. It goes beyond computing with only selected keys. It reduces long-context processing costs by jointly optimizing key selection and the stages that read and compute with those keys. The guidance targets model designs that incorporate sparsity during training.

1. Reducing computation makes key selection a new bottleneck

Standard attention compares each query against all keys, while sparse attention uses only a few thousand highly relevant keys. It is like opening only the pages you need rather than reading an entire book. However, the indexer that finds those pages still scores every key.

According to the company, the indexer and top-key selection accounted for 47% of a layer's runtime when DeepSeek V3.2 processed a 128K-token input. Finding keys took longer than the actual attention computation, shifting the optimization focus to selection.

NVIDIA GPU, DeepSeek V3.2 DSA, input length of 128K tokens, based on per-layer input processing time

Time shares of major stages per layer in DeepSeek V3.2 (%)Indexer and top-key selection: Time share 47; Core attention: Time share 24; Feed-forward network computation: Time share 25Indexer and top-key selection47Core attention24Feed-forward network computation25
Time shares of major stages per layer in DeepSeek V3.2 (%)
Time shares of major stages in a sparse attention model (%)
StageTime share
Indexer and top-key selection47
Core attention24
Feed-forward network computation25

The table shows the shares of time spent on major stages for the same model and input length. According to the company, the share for the indexer and top-key selection exceeded the 24% for core attention.

2. Input processing and token generation have different bottlenecks

The indexer compares queries and keys using matrix multiplication, an operation that multiplies and sums rows and columns. Postprocessing then produces one score per key. When processing the input all at once, the computational cost of large matrix multiplications matters. During next-token generation, even a few queries must read the entire key cache, the memory storing previously computed keys. Memory reads therefore become the main bottleneck.

The company's baseline indexer measurements below use an 8-bit floating-point (FP8) kernel, a program that performs computations on the GPU. They use 64K indexer keys, a compression ratio of 4, and a head dimension of 128. The compression ratio describes the reduction from the original, and the head dimension is the vector size handled by each head. There is one key head, with batch sizes of 1 for input processing and 128 for token generation. The compression-ratio experiment fixes the sequence length at 256K.

Group size is the number of query heads sharing one key head. According to the company, a 16x increase in group size changed token-generation runtime by 2%. This is because the same key cache is read once and used by multiple query heads.

The benefits and costs differ by stage. Larger groups make greater use of keys read from memory, but increase computation and runtime during input processing. Model developers must weigh the cost of initially processing long inputs, not just token-generation efficiency.

3. Fit draft verification to the tile capacity

Speculative decoding, which verifies multiple draft tokens together, also reuses the same key cache. Here, group size and the number of queries to verify share space in a tile, a unit of data the GPU processes together. NVIDIA said that 32-bit floating-point (FP32) postprocessing can practically fit about 200 query rows in one tile.

According to the company, processing 6 queries at a group size of 32 increased the workload to 6x and runtime to 1.37x. At a group size of 8, using 26 queries increased the workload to 26x and runtime to 1.87x. Each figure is relative to processing one query at the same group size. Exceeding the tile's capacity requires additional cache reads, causing runtime to jump.

NVIDIA recommended indexer head dimensions of 128 or 256. According to the company, increasing the dimension from 64 to 128 raised input processing time to 1.1–1.2x. During token generation, read costs rose with the additional key data. Increasing key heads from one to 8 raised input processing time to 1.3–1.6x and token-generation time to 7.6x.

4. Reduce both scoring targets and intermediate data movement

Compressing keys to represent multiple tokens, narrowing candidates hierarchically, or sharing selections across layers can reduce scoring work. In NVIDIA's compression-ratio experiment, halving the compression ratio doubled input processing time. Token-generation time also roughly doubled, reaching 1.8x with an 8K key cache.

Fusing Top-k, which retains the highest-scoring keys, with the indexer kernel can eliminate writing all scores to memory and reading them back. TensorRT LLM also offers a method that reuses the previous step's selection as an estimate when generating the next token.

According to the company, this reuse method was 2.1–2.4x faster during token generation in DeepSeek V3.2 and DeepSeek V4 than the existing selection method, which filters scores by digit.

Developers can refer to FlashInfer's public implementation for an approach that combines indexing and selection in one kernel.

5. Selecting contiguous blocks makes it easier to share reads

Selecting keys token by token scatters the selected data throughout the cache. Selecting contiguous blocks, units of adjacent data, enables grouped reads and data sharing among queries that select the same block. This is why NVIDIA recommends considering large blocks first.

According to the company, in an experiment with 2,048 selected keys per query held constant, halving block size nearly doubled input processing and token-generation times. Even with the same computation and total key-value data, more blocks mean more work loading query data and combining partial results.

The unit of optimization for long-context inference is not an individual attention kernel, but the entire path from key selection to data reads.