llama.cpp now supports reference-based voice cloning in the Drama 3 speech synthesis model. Developers can specify a voice with a reference clip and transcript, after reconverting the existing auxiliary audio model file.

1. Specify a voice using a reference clip

b12590 was published on November 20, 2026 as a prerelease, a test version before the official release. Previously, Drama 3 ignored voice-cloning reference files with a warning. It now feeds clip audio into generation to reflect the voice.

The transcript contains the words spoken in the reference clip. Without a speaker label, it is treated as speaker 0. If the text to generate has no speaker label, it inherits the label from the reference transcript.

2. Add an input path to the output codec

The key is to use the output codec for input too. Drama 3's multimodal file handles multiple types of information and holds the speech generator and codec. It now includes an encoder that turns reference audio into codes the model can read, alongside the output decoder.

Encoder conversion code is shared with Fish S2. The PR author says Fish S2 conversion output is unchanged, retaining 511 tensors, multidimensional numerical arrays, and 20 metadata entries describing file structure and properties. Drama 3 gains an input path while existing Fish S2 conversion output stays unchanged.

3. Use it from the command line

The following example comes from the PR author. It specifies the model, multimodal file, reference audio, transcript, and new text to speak.

llama-tts -m drama3.gguf -mm mmproj-drama3.gguf \
--tts-speaker-file reference.wav \
--tts-speaker-text "<|speaker:0|> transcript of the clip" \
-p "text to speak" -o out.wav

Only one reference audio input is allowed per request, and only its first 30 seconds are used. Older auxiliary files without an encoder trigger a warning and ignore reference audio, so reconvert the multimodal file before using voice cloning.

4. Test results reported by the contributor

The PR author tested Drama 3 checkpoint 78e5ab79, a model file storing a training state. The core generator used 2 bits per weight and 192 experts. The multimodal file used Q8_0 quantization, a technique that reduces stored numerical precision.

The author tested on CUDA and reported WavLM speaker similarity to the reference clip of 0.883 with reference audio and 0.563 without it. Using reference audio yielded higher similarity against the same clip.

Of 91 single-reference audio prompts, or model instructions, in held-out validation data, 88 passed the author's speaker verification test. Average similarity was 0.956.

Content was checked through speech recognition. The author reported that 136 of 138 prompts met the per-category word error rate thresholds, measuring word-level recognition errors. This checked spoken content as well as voice similarity.

5. More file and input processing instead of another model

Reference support adds a codec encoder to the multimodal file and a prompt row for each input audio frame, a short segment of audio data. Files must be rebuilt, and the model processes more input. In return, one reference clip specifies speaker traits without adding a separate generator model.

The practical change is that reusing the Fish S2 codec lets reference voices enter the existing Drama 3 generation workflow after reconverting the auxiliary file.