NVIDIA's large language model inference tool has expanded model support and removed the cache manager's Python implementation. Developers gain more model choices, but services relying on the old implementation must review settings and event collection.

1. Model support and prerequisites

Released on October 14, 2026, this prerelease adds GLM-5.3-Flash to the PyTorch backend. It supports text, image, and video inputs. It uses the official 8-bit floating-point model weights as provided.

The model support proposal's deployment guide specifies 4 B200 GPUs for unified serving and 8 for disaggregated serving, which separates input processing and output into distinct workers. It requires Linux and a CUDA driver of 575 or later.

Text-only services work with the bundled Transformers library. Image and video inputs require the following installation.

  pip install "transformers==5.17.0"

The container also needs fla-core and einops installed.

Video decoding additionally requires opencv-python-headless.

2. Read speed figures within their test conditions

Multi-token prediction proposes several upcoming tokens in advance. The deployment guide in model support PR #19136 reports measurements using TensorRT LLM 1.3.0rc29 and 4 B200 GPUs.

The tests used 8-bit FP8 weights and a 16-bit BF16 cache, with input and output lengths of 1,024 tokens each. Random input used seed 0, selected the most probable token, and ignored end tokens. Warmup used twice the concurrency in requests, followed by 2 measurement runs with 5 times as many requests. The predicted-token acceptance rate was not artificially fixed.

At 1 concurrent request, model computation and expert modules were each distributed across 4 GPUs. Experts are subnetworks activated selectively based on the input. Predicting 3 tokens yielded 389 tokens per second per user and 92 per GPU. With prediction off under the same conditions, mean time per token was 5.57 milliseconds and speed was 180 tokens per second per user. Single-user generation speed differed by about 2.2 times.

At 256 concurrent requests, attention requests were split across 4 GPU processes, with expert modules also spread across 4 GPUs. Attention determines which parts of the input to reference. Predicting 1 token achieved 2,735 tokens per second per GPU, with a batch limit of 64 per process.

Per-user figures exclude time to first token, while per-GPU figures include input processing and the first-token wait. These results reflect the tested configurations and concurrency range, not proof of maximum throughput.

On 1,319 math problems in reasoning mode, selecting the most probable token with an output budget of 4,096 tokens produced an accuracy gap within 0.08 percentage points of the Hugging Face implementation.

3. Cache management reduced to one implementation

The KV cache stores computation results for previously processed tokens. PR #19154 removed the Python implementation from KVCacheManagerV2, which manages this cache. Only the implementation that was already the default remains, reducing the burden of keeping two sets of key computation, eviction, and statistics logic in sync.

The package imported from Python has not disappeared. The remaining package exposes bindings to the C++ implementation. The backend selection environment variable and the build option for accelerating the Python implementation were also removed. Deployment and build procedures that specify them need updating.

4. Distinguish when event collection changes occurred

The removal PR also deleted the Python streaming path that emitted cache changes immediately. In that change, kv_cache_config.kv_events_config is rejected at startup. The alternative collects events via kv_cache_config.event_buffer_max_size and retrieves them per iteration.

However, the final release list separately includes C++ streaming event support in PR #19359. The removal PR alone does not establish that this version lacks all streaming support. The supplied materials do not explain compatibility between existing settings and the new path.

5. Adoption benefits and migration costs

Cache precision is also a choice. An 8-bit floating-point cache halves storage for compressed key-value representations. However, selected cache rows are converted to compute precision, moved in small batches to temporary storage, and passed to attention repeatedly. This extra work can increase time to first token and reduce input throughput for large inputs. Index precision and recurrent-state precision remain unchanged.

Reusing computation from the model's earlier stages requires a policy for periodically saving intermediate states.

GLM-5.3-Flash does not support beam search, which retains multiple candidates while selecting an answer.

The release also lists a known issue on H100 with tensor parallelism of 2, which splits model computation across 2 GPUs. During streaming and remote execution, reusing an interprocess communication worker pool across model objects can cause initialization to hang.

The saved official release notes cover only through 1.3, so the release list and individual PRs must be read together.

This version centers on broader model choices and a simpler implementation, but adoption decisions should also depend on verifying that existing cache settings can be migrated.