llama.cpp v0.9.0: Unified inference improvements, broader speech and model support

1. Improved cache paths for multiple requests

The new flash_attn_ext_rows operation improves unified KV cache use across requests. This cache stores each request's keys and values. Each query segment refers to its assigned cache rows. The project also reports restored prefill performance on paths without a unified cache. Prefill computes the input context first.

Quantization gains OCP FP8 E4M3 format and conversion support, a specification for representing numbers with fewer bits. Quantization expresses model values in smaller formats; this update expands the available formats. The public headers also include a value identifying the new format.

Gemma 4 E2B and E4B receive computation graph optimizations that streamline operation relationships. The release notes report major prefill improvements for both. The changes address model-specific input processing as well as shared paths for multiple requests.

2. Support for four new model architectures

Newly supported models include Clover-35B-A3B-Preview and Limite 1B. The former is supported in autoregressive mode, which generates subsequent content based on previously generated content.

Metis-1.6 and Mistral Large 4 are also added. Metis-1.6 combines Mamba-2 and GQA attention in two variants: Praxis at about 2.9B and Logos at about 10.2B. Alongside cache and computation improvements, the tool expands its supported model architectures by four.

3. Expanded speech generation and multilingual transcription

Speech generation gains Chatterbox TTS support. Pocket TTS adds predefined voices without voice cloning. Users can now choose a prepared voice instead of cloning their own.

Nemotron 3.5 Transcribe supports offline transcription in 40 languages. According to the project, it incorporates prompt context and associates utterances with speakers. This release expands both speech generation and the conversion of recordings to text.

For video input, qwen3-vl prompt format support is improved. Gemma 4 image inputs are now resized without padding. Model-specific input improvements thus extend beyond speech.

4. Discover voices on the server and select them by name

The server's GET /props response lists the model's predefined voices. Speech generation requests can specify a voice name. Service developers can now connect voice discovery and selection through the server API.

/v1/responses now merges additional parameters into converted chat responses. Integration code using the existing response path should be checked for this change.

The web interface keeps the conversation at the bottom until the user scrolls up. Tool calls without responses also show an interrupted status. These changes refine how the interface displays conversation progress.

5. Broader uses and the burden of interface migration

A general batched tensor copy API copies multidimensional numeric arrays together. Implemented in Metal and CUDA, it is intended to replace the existing _2d family. Code calling the old family directly may need changes during migration. The new functionality brings both benefits and integration maintenance costs.

The underlying ggml library moves to v0.29.0. It adds initial tensor parallelism support, splitting model computation across nodes, and optimizations for computation backends, the software that performs the calculations. Service developers must review both new models and the computation paths they actually use.

The release notes provide no measured performance gains or hardware and configuration details. They also name no next version: the string ‘1.0.0’ appears nowhere in the notes, and no milestone or preparation stage for such a version is described. The only versioning pointer is a ‘More info’ link to the ggml discussion ‘Releases and versioning of ggml-org projects’, whose own contents are not verifiable from the release notes.

v0.9.0 expands concurrent inference paths and model and speech capabilities, making interface migration checks important for existing services.