ExecuTorch 1.6.0: Torch-free runtime, batched LLM serving

1. Reducing PyTorch dependencies in model execution environments

Released on November 12, 2026, this version's Python runtime runs exported .pte model files without PyTorch. Separating model preparation from execution environments allows deployments to require fewer dependencies.

Inputs can be passed as NumPy arrays, buffers, or DLPack, a standard for exchanging arrays between frameworks. Loading the runtime no longer also loads model conversion and compilation modules, and test-only packages have been removed from installation dependencies. These changes directly affect developers setting up model execution environments.

2. Batching requests and reusing computations for shared prefixes

Batched serving and prefix caching have been added to language model execution on CUDA and MLX. Alongside processing multiple requests together, they reuse computation results for shared request prefixes. Service operators should examine changes not just to individual operations but also to request processing flows.

CUDA adds a batched executor, prefix cache reuse, and on-device sampling that selects output tokens on the GPU. The KV cache, which stores information about previously processed tokens, is managed outside CUDA graphs, which group GPU work for execution. Both request batching and cache management have changed.

DFlash speculative decoding, which proposes next-token candidates in advance, is also supported. MLX adds a batched KV cache to support it, and the release notes say the related GPU language model features are fully integrated into Muse Glimmer.

3. GPU autotuning also targets small batches

CUDA and ROCm add INT4, INT5, INT6, and INT8 quantized matrix multiplication kernels. These kernels, code that performs GPU computations using reduced-precision values, are autotuned for token generation and small-batch input processing. The release notes specify M ≤ 64 for small-batch input processing.

Autotuning tests candidate operations to select an implementation. This version measures candidates inside CUDA graphs so CPU overhead does not influence selection. Developers can also specify example inputs for benchmarking, enabling tuning that considers the inputs they will actually use.

Long-context attention, the computation that focuses on important information in context, has also been improved. In-kernel causal masking, which prevents references to future tokens, and hardware-aware split decoding revise how computations reference earlier tokens and how work is divided.

4. Expanding execution streams and model loading paths

The experimental executorch.runtime.cuda module allows CUDA stream selection from Python. It provides a control point for developers who want to specify GPU execution flows from Python.

Another experimental feature, ExecuTorch Native, introduces portable .ptn programs that can be used across execution environments. Vulkan or ggml engines transform programs for execution at load time. This approach moves some work to adapt model files to their execution environments into the loading stage.

5. Fewer deployment dependencies, more environment migration work

Python support now spans 3.11 through 3.15. CUDA wheels, distribution package files installed in Python, add 13.4 and drop 12.6 and 13.0. New execution environments need to account for this support range in their installation plans.

The benefit is more options for building PyTorch-free execution environments and expanding GPU request processing paths. However, environments relying on Python 3.10 or removed CUDA wheels need to migrate to supported versions, and Jetson Orin requires building from source. This update calls for reviewing both reduced dependencies and changes to existing deployment procedures.

ExecuTorch 1.6.0 expands options for leaner execution environments while strengthening GPU request processing and cache reuse.