Support for two speech recognition models gives developers more options for converting audio to text in llama.cpp. They work with file uploads or real-time audio input, and one also outputs speech timing and speaker information.

1. Support spans model conversion and server requests

Released on November 15, 2026, b12426 is a prerelease, a test version before a full release. It adds two models to mtmd, which handles audio, images, and other inputs. The changes span model conversion to a runnable format and server-side audio handling.

The supported models are bosonai's Higgs Audio v3 and microsoft's VibeVoice-ASR. The former handles speech transcription, while the latter produces transcripts showing who said what and when.

Reviewer ServeurpersoCom tested Q8_0 text model weights, an 8-bit storage format, with a 16-bit floating-point audio projector model. Tests used English and French clips and covered the transcription API, chat completions API, and real-time transcription sessions.

2. Higgs Audio v3 splits audio into 4-second segments

Higgs Audio v3 splits audio into 4-second segments and independently generates acoustic features, numerical representations of sound, for each segment. Under the stated test conditions, the reviewer reported accurate English and French transcription. A 34-second English clip was also transcribed accurately after being split into 9 segments.

For real-time input, each segment is processed once full, and any short remainder is processed separately when input ends. Rather than accepting a long recording only as a single unit, it splits incoming audio for sequential processing.

3. VibeVoice-ASR outputs timing and speaker information

VibeVoice-ASR directly reads a 24 kHz mono waveform, which represents changes in sound over time. By default, it splits files into 30-second segments while also reading context from preceding segments. Unlike the previous model's independent processing, it connects previously heard audio to the next segment.

Output is JSON containing start and end times, speaker IDs, and content. This gives developers a way to locate speech segments in transcripts or handle content by speaker.

The prompt requires the audio duration. The server obtains it from the audio file's metadata and includes it in the request. Names or specific words can also be supplied as additional context. The following source excerpt inserts duration and extra information into the request text.

        } else if (is_vibevoice_asr_template(src)) {
            // without the audio length the model does not know where the audio ends and loops
            const char * keys = "transcribe it with these keys: Start time, End time, Speaker ID, Content";
            if (context.empty()) {
                asr_preset.user = string_format("\nThis is a %.2f seconds audio, please %s", audio_length, keys);
            } else {
                asr_preset.user = string_format("\nThis is a %.2f seconds audio, with extra info: %s\n\nPlease %s", audio_length, context.c_str(), keys);
                asr_preset.has_context = true;
            }
            asr_preset.media_first        = true;

The reviewer reported that a 36-second clip was split at the 30-second boundary with accurate timestamps and speaker information. With the same prompt, a comparison clip matched the transformers implementation word for word. Additional information supplied in the prompt was also reflected in the result.

4. Shared server paths, model-specific audio processing

Developers adopting these models can use model conversion support and the server's audio input paths. The review covered /v1/audio/transcriptions, /v1/chat/completions, and real-time transcription sessions.

While the same server offers more speech recognition options, the processing layer responsible for each software stage differs by model. In particular, VibeVoice-ASR's real-time processing retains information from preceding segments as state.

VibeVoice-ASR's streaming volume normalization adjusts loudness using audio received so far, not the entire clip. Neither model's streaming transcription implementation accepts an input prompt. Additional context in file requests remains available separately from this streaming limitation.

b12426 goes beyond adding speech recognition models by offering more ways to use transcription, timing, and speaker information through existing multimodal server paths that handle multiple data types.