ZONOS3-Mini: Real-time voice cloning, multiple runtime support

1. Targeting on-device execution by computing only what is needed

The model activates about 100 million of its 800 million parameters when generating a token. Its mixture-of-experts architecture selects only the necessary parts, using a subset of a large model to generate speech.

According to the model card, it supports 43 languages, including Korean, and its weights are licensed under Apache 2.0.

The company says the earlier ZONOS2 used 8 billion parameters and an 86-frame-per-second codec, limiting deployment. The new version uses its ZAC-1 codec to encode sound at 15 frames per second and convert it to 44.1kHz audio.

The company reports that, under similar data conditions, a ZAC-1 model had a 10x lower word error rate than a DAC model. This is the share of words misrecognized in speech recognition. Acoustic similarity was also higher. The company presents this as improving pronunciation accuracy while reducing the sound representations to process.

2. Browser responsiveness and server processing capacity

The company reports browser time to first audio with WebGPU of under 570 milliseconds on Pixel 9 and under 150 milliseconds on M3 MacBook Air. The aim is to keep the wait for speech generation short on phones and laptops.

The model card reports company server tests on a single NVIDIA H100 80GB, using a representation summarizing the speaker's voice as input. With 8 concurrent requests, median time to first audio was 80 milliseconds. The median is the middle value when measurements are sorted. Peak throughput was 457 seconds of audio per second at each model's optimal concurrency, or number of simultaneous tasks. This shows capacity for handling multiple requests separately from response startup speed.

The company also released its evaluation framework. ZTTS2-Eval covers read speech in 35 languages and real-world recordings in 20 languages. It was used to build a metric comprising 420 items. ZONOS3-Mini ranked first among open models in this company evaluation.

According to the company, Fish S2 Pro ranked next, followed by MOSS TTS v1.5. The evaluation considers voice similarity, rhythm, and pitch as well as accuracy in each language.

3. Choosing a path from browser trials to server integration

In a WebGPU-enabled browser, the model runs on the user's device without a separate installation or server. Downloaded models stay in the browser cache for subsequent visits.

Zonos Voice Studio is available for voice production, alongside Lite for quick voice cloning.

On servers, SGLang-Omni supports an OpenAI-compatible voice API and streaming, and can clone reference speech without a transcript.

The company's native runtime offers CUDA for NVIDIA graphics cards and MLX for Apple chips.

A C++/ggml-based server and command-line tools are also available, and the native runtimes share a common HTTP API. Developers can use a common integration approach even when the execution environment changes.

The browser option reduces the burden of setting up a separate server. In return, the model must be downloaded on the first visit, and the user's graphics processing unit handles computation.

The following Python library example loads reference audio, creates speaker information, and generates speech.

import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
model = Zonos.from_pretrained("Zyphra/ZONOS3-Mini", device="cuda")
wav, sr = torchaudio.load("speaker.wav")           # 5-10s reference clip
speaker = model.make_speaker_embedding(wav, sr)
cond  = make_cond_dict(text="Hello, world!", speaker=speaker, language="en-us")
codes = model.generate(model.prepare_conditioning(cond))
audio = model.autoencoder.decode(codes)[0].cpu()
torchaudio.save("sample.wav", audio, model.autoencoder.sampling_rate)

4. Adjusting both the voice and word-level delivery

Users can adjust voice traits such as age, gender, accent, and timbre, along with speed, emotion, and energy. They can also change vocal projection, from speaking close by to calling across a room.

Word-level delivery can be controlled during generation, and an editing model can revise sentences afterward. For game characters or narration, this can help refine the performance of specific phrases after choosing a voice.

The company says control values are generation targets, not measurements of actual age or emotion. Changing speed or energy may also change volume, pitch, and duration. Breathy texture is approximated with a soft timbre rather than a dedicated control.

ZONOS3-Mini's key contribution is bringing real-time voice cloning closer to developers' deployment environments through a small active model and multiple execution options.