LLM inference engine vLLM has released a version that enables cache expansion by default to use more free GPU memory and unifies GPU model execution paths. Model service developers gain room for concurrent requests and fewer executor choices to manage during deployment.
1. Expand the cache using actual free memory, not estimates
On CUDA and ROCm, the new method sizes the cache using actual free memory after warmup and graph capture, which records GPU execution sequences in advance. It replaces estimated reservations with measurement after preparation.
According to the vLLM project, the MiniMax-M3 cache on H200 grew from 433,000 to 672,000 tokens. The project says a larger cache boosts throughput, or work per unit of time, and reduces time to first token under high concurrency.
When extensible caching is enabled and the user has not set a value, GPU memory utilization defaults to 1.0. Options to retain the previous cache sizing method are listed in the upgrade section below.
2. Unify GPU execution under the second version
vLLM 0.34.0 removes the first version of Model Runner, which runs models on GPUs, leaving only the second version. Adapting deployment settings to the new runner is a key upgrade task.
Existing logits processor code, which adjusts scores for next-output candidates, must move to vllm.v1.worker.gpu.sample.logits_processor.
The feature that overlaps execution of two batches also works with the new runner. It can still be combined with speculative decoding, which proposes upcoming output in advance, after runner unification.
3. Hardware-specific improvements for long inputs and output generation
The decode stage, which generates output tokens, also improved. The project reports up to 1.95x decode speed for Step5 on 4 GB200 GPUs using TRTLLM-Gen. This figure applies to Step5's NVFP4 MoE running on TRTLLM-Gen.
For GLM-5.3-Flash on NVIDIA hardware, the project reports 1.6x output throughput with DeepGEMM FP8 MegaMoE, an expert-model computation path, compared with DEP4.
For long inputs, the path that selectively attends to needed parts and splits input processing by token has improved. It is enabled by default for 1024 or more input tokens under tensor parallelism, which splits model computation across GPUs.
The project reports 1.24–1.40x faster time to first token for GLM-5.3 on GB200 with tensor parallelism and prompts of 32K–128K tokens.
Optional output optimizations are also available. The project reports 1.34–1.59x decode attention performance for Kimi-K3 on GB300 with an attention path that balances cache work while attending to input information.
The project reports 1.5x single-stream decode speed for Muse Spark Open on GB300 using an optional specialized path.
The project reports 3.0–8.8% lower inter-token latency for Qwen3.5 on NVIDIA hardware using interruptible CUDA graphs without torch.compile.
On AMD, the project reports 35–57% better interactivity at 1–8 concurrent requests for Muse Spark Open on 8 MI355X GPUs with DSpark speculative decoding and an optimization that fuses output-generation operations.
4. Choose between new defaults and previous settings
The default cache block size changes from 16 to 64. Backends with a preferred size take precedence, while ROCm paged attention, which manages memory in blocks, still uses 16. Explicit user block sizes also take precedence.
The release notes provide this default CUDA installation command:
pip install vllmTo retain previous cache sizing, set memory utilization to 0.9 or disable extensible caching. The first two lines below show these alternatives; the last is a separate option for the previous block size.
--gpu-memory-utilization 0.9
--no-enable-extensible-kv-cache
--block-size 16AMD distributions use wheels, which are installation packages, and images for ROCm 10.0 instead of 7.2.
5. Weigh concurrency gains alongside pre-deployment checks
Allocating more memory to the cache creates room for more requests. The trade-off, according to the vLLM project, is a possible 6–18% increase in per-token decode latency for some H200 models at 128 concurrent requests.
Before deployment, remove environment variable settings that select the old runner. Leaving VLLM_USE_V2_MODEL_RUNNER=0 causes startup to fail.
Evaluate both concurrent request capacity and output intervals per request. For long-input services, focus on time to first token; for long-output services, examine generation speed and latency changes on the relevant hardware.
vLLM 0.34.0 centers on cache expansion using actual free memory, and operators should choose settings by weighing concurrent throughput gains against changes in decode latency.
