Xinference 4.2.0: Prefix cache sharing, rolling reloads

1. Replicas reuse the same input prefix

Prefix cache sharing reuses previously computed results instead of recalculating the same input prefix on each replica. Xavier handles this sharing, focusing on reducing repeated work during prefill, the initial input-processing stage.

Replicas running vLLM and SGLang are eligible for sharing. This enables computation results to be reused in environments running multiple replicas with these inference engines.

Replicas running TRT-LLM and MLX are also eligible. Extending sharing to environments using multiple inference engines is a key part of this change.

While an agent waits for a tool response, it can prioritize retaining KV prefixes, the computation results for input prefixes it will use next. This aims to reduce the burden of reprocessing the same input when work resumes.

On the storage side, LMCache adds Mooncake Store support. Lifecycle management for retaining and reusing agent caches also extends to tiered storage that uses multiple storage layers.

2. Reload replicas one at a time while retaining weights

Rolling reloads reload replicas one at a time while retaining GPU weights, the numerical information learned by the model. Requests in progress finish on the previous engine instance while replicas are replaced. Replicas are reloaded in turn, leaving the previous engine available to finish existing requests.

3. Update the software in sequence through the web UI

Administrators can check for and install Xinference updates through the web UI. Alongside model reloads, this provides an update path for the software itself, allowing version management to start from the operations interface.

Upgrades proceed through workers, the processes that execute tasks, then supervisors, the components that manage execution units, and finally the API. While rolling model reloads handle running models, this procedure updates the programs that make up the inference service in sequence.

4. Examine throughput and latency on running models

The web UI also adds a way to start benchmarks on running chat models. Users can vary concurrent request counts and compare throughput and latency across runs. Operators can examine how their models behave under different load levels.

Comparison metrics include time to first token and time per output token. Results can be exported as JSON, allowing users to take benchmark records out of the UI. While cache sharing reduces repeated computation, this feature offers a way to examine the processing performance of running models.

5. Retain computation results to reduce recomputation

Cache sharing reduces the burden of reprocessing the same input. In exchange, operational focus shifts to storage management for retaining and reusing computation results. Rolling reloads also retain GPU weights while the previous engine finishes ongoing requests, requiring consideration of both request continuity and resource retention.

Cache sharing applies to compatible prefixes, while weight-preserving rolling reloads target supported vLLM and SGLang deployments. The release notes do not specify performance gains or the additional resources needed to retain weights.

Xinference 4.2.0 focuses on easing model-service operations by reusing computation results across replicas and replacing replicas in sequence while accounting for ongoing requests.