Xinference 4.1.0: TRT-LLM integration, stronger inference server security

1. Connecting a new inference engine to the same service interface

TensorRT-LLM is now an inference backend that runs model inference. The unified API handles supported models' chat, generation, and streaming of results as they appear. Developers can choose engines without changing the service interface.

Compatible models can also hand off work between different inference engines. The release notes cite deployments combining NVIDIA GPUs and Apple silicon as an example.

Xavier's sharing of intermediate computation results for common input prefixes has also been extended to TRT-LLM.

The new engine can split input processing and sequential answer generation into separate stages. This feature can use either the default approach or Xavier.

There are also more places to store shared computation results. The optional LMCache storage backend allows caching on disk as well as in memory. The new engine supports this storage method too.

2. Replacing processes after ongoing requests finish

Live upgrades update supported model engines and replace server worker, API, and management processes. Ongoing requests and streams finish before old service processes exit. The web UI controls single-system and distributed upgrades.

This feature targets POSIX environments and remains experimental.

Settings for context length, the input range a model can reference at once, are now unified across inference engines. A web UI slider adjusts the limit, with validation based on the effective model configuration.

The release also adds memory estimation for launching models across multiple inference engines.

3. Stronger trust boundaries for model loading and administrator creation

This is the first patched release for four published security advisories. The project recommends that users of affected versions update to 4.1.0 or later.

A critical CosyVoice model-loading issue is addressed by requiring explicit operator trust approval. This blocks unapproved custom sources or explicit local paths from reaching the loader, the component that loads model files.

Creating the first administrator requires an operator setup token. This prevents unauthenticated callers from taking over an uninitialized server. The vulnerability affects versions from 3.0.0 up to, but not including, 4.1.0.

Paul White of Cairn Security reported this issue and verified the API-level fix.

Automatic model registration reads configuration files only from approved paths. Extra model launchers require an operator allowlist. These changes address reported configuration exposure, filesystem probing, and arbitrary module imports.

Chat error responses hide internal diagnostic details, including addresses and process identifiers of workers, the separate processes that perform tasks.

The release also improves validation of shared-store API keys against their current state.

4. Operators configure trust approvals and allowlists

On new instances, the first administrator is created with the setup token shown in API server startup logs. Custom CosyVoice sources require worker-side trust settings, and the sources themselves must be trustworthy.

Extra model registration paths and launchers, tools that specify how models run, each need allowlists that admit only approved entries. The block below lists the related environment variables in the release notes.

XINFERENCE_TRUST_REMOTE_CODE=1
XINFERENCE_MODEL_REGISTER_ROOTS
XINFERENCE_FLEXIBLE_LAUNCHERS
XOSCAR_AUTH_KEY

Authentication for Xoscar, which handles inter-process communication, is opt-in. To use it, set the same nonempty authentication key on every participating process.

The benefits are more engine and cache storage choices, plus finishing existing requests during updates. Operators must directly manage the first-administrator token, trusted sources, and allowed paths and launchers.

5. Fewer repeated scans of tool-call streams

Tool-call streaming no longer rescans strings whenever a new chunk arrives. The improvement covers tool-call streams from multiple model families, including DeepSeek and GLM-5.

According to the company, processing a 32KB tool call fell from 12.5 seconds to 11 milliseconds. This result concerns reduced repeated scanning of tool-call streams, illustrating a specific processing path improved in this release.

Xinference 4.1.0 focuses on expanding inference engine choices without changing the service interface, while placing sensitive server operations under explicit operator control.