TensorRT-LLM is NVIDIA software for running large language model inference and serving requests. Version 1.3.0 unifies execution on PyTorch and serves image and video generation through the same tools. Developers connecting models to services gain options for generative models beyond text inference.

1. Unified PyTorch execution replaces engine builds

This version removes the infrastructure for building engines in advance and unifies execution. Engine build commands and related configuration objects are also gone, leaving only llmapi among Triton backend models.

The existing C++ executor API has also been removed. The legacy runtime API for execution environments and the batch manager API for grouping requests are gone too. Deployment automation built for existing engines or the C++ executor must migrate to PyTorch execution.

Only TorchSampler, which requires FlashInfer, is supported for selecting the next token. This combination replaces the choice of previous samplers.

TensorRT and ONNX are no longer mandatory dependencies, reducing the external components needed for installation and execution.

According to the company, asynchronous input preparation and the removal of synchronization in beam search, which explores candidate sequences separately, reduce CPU overhead.

According to the company, prefetching checkpoint data reduces startup time. This is distinct from improvements that reduce processing overhead during execution.

2. Broader support for new models and existing GPUs

DeepSeek-V4 and Kimi K3 have been added to the support list for Blackwell and preview-stage Rubin. Qwen3.5 also supports Hopper, offering a path for existing GPUs. Kimi K3 also supports multimodal input and multi-node serving.

Rubin support is available as a preview validated on Vera Rubin. It supports testing deployments for next-generation hardware.

Quantization reduces the precision of numerical representations. This release supports W4A16 NVFP4 through Marlin, storing only weights, the numerical information learned by the model, at low precision. This path also runs NVFP4 weight checkpoints on Hopper and Ada Lovelace. Unlike the preview for new hardware, it offers a way to use NVFP4 checkpoints on existing GPUs.

MoE, which selects some of a model's subnetworks for computation, supports up to 2048 experts.

Separately, a new feature returns which experts handled computation for each token. This information can be used in reinforcement learning, which learns behavior based on rewards.

3. More options for API access and disaggregated serving

The Anthropic Messages API has been added, and the Responses API has been expanded. The OpenAI-compatible API allows each request to specify its priority and the function to call.

A gRPC remote call server and a multiprocess HTTP frontend for receiving and forwarding requests have also been added.

Disaggregated serving, which separates input processing from output generation, adds generation-first scheduling and routing that groups requests from the same conversation. A KV cache stores information computed from earlier inputs for reuse in later generation. This version also adds per-conversation cache block reuse. API expansion broadens service access interfaces. Per-conversation routing and block reuse affect internal handling by grouping ongoing conversation requests and reusing earlier computations.

Inputs with a common beginning are handled by prefix-aware scheduling, which is enabled by default. It avoids duplicate processing of shared prefixes, changing repeated-input handling as well as the available APIs.

Encoder-only models, which use only the component that converts inputs into vectors, gain an embedding API and a fast llm.encode() path. Services providing search embeddings are also covered by this expansion.

4. Image and video generation in the same serving framework

VisualGen is a set of features for image and video generation models, served alongside LLM inference through trtllm-serve. New endpoints cover image generation, image editing, and video generation. Services can configure endpoints for their generation tasks while using the same serving tools.

Added image models include FLUX.1 and FLUX.2, while video models include Wan 2.2 and LTX-2. Services can expose endpoints for image editing and video generation as well as text responses, broadening their scope.

5. Check configuration precedence and defaults before upgrading

The TensorRT backend has been removed, leaving PyTorch as the sole execution backend. TensorRT engine-based models have also been removed from Triton, where only the llmapi model remains available.

Explicit flags in serving, evaluation, and benchmark commands, which specify values at invocation, now override YAML settings. Even deployments reusing the same configuration file need to check command-line values first.

KV cache transfers now have a default timeout of 60 seconds, whereas previously there was no timeout. When transfers are delayed, request processing now applies a fixed waiting limit instead of waiting indefinitely.

Separate from this transfer timeout, request settings raise an error if logprobs or prompt_logprobs exceeds 100. This is an upper limit on allowed request settings, not a processing speed limit.

Per-request chat templates must be explicitly allowed before use.

Usage telemetry, which collects and sends usage information, is enabled by default. The following disabling methods are listed in the original source; choose one.

TRTLLM_NO_USAGE_STATS=1
DO_NOT_TRACK=1
--no-telemetry
telemetry_config=TelemetryConfig(disabled=True)

Known issues include possible initialization failures for FLUX.1 checkpoints exported with NVIDIA Model Optimizer using static quantization, which lowers precision during conversion. The Responses API also retains compatibility gaps for some input formats, conversation continuation, and tool calls.

Adoption decisions for this release should start with how easily existing deployments can accommodate the PyTorch transition, rather than the expanded model list.