A speech synthesis model has been released that clones voices from short recordings and transcripts and inserts laughter or sighs into dialogue. Lure TTS also provides an engine that emits audio before generation finishes. This lets voice content developers build services combining reference voices with acting directions.
1. Voice cloning from short reference clips
Lure is an English-only text-to-speech model with 600 million parameters. Parameters are numerical values adjusted during training. Given a 3–12-second reference clip and its transcript, it generates new sentences in that voice. Output is 24kHz mono audio with a single sound channel.
On the model card's Seed-TTS WER word error metric, Lure (600 million parameters) scores 1.51 and Fish Drama 3 (35 billion parameters, mixture-of-experts architecture) scores 1.54. On LibriSpeech-PC UTMOS, which assesses naturalness, Lure leads the comparison table at 4.43, versus 4.41 for Qwen3-TTS (600 million–1.7 billion parameters). The developers generated each sentence once using official evaluation scripts and default sampling, the method that determines generated outputs. Comparison figures were reported by the respective developers.
2. Effects and acting directions within speech
Users insert parenthesized tags wherever a sound is needed in dialogue. For example, (sigh) indicates a sigh and (sharp inhale) a sharp intake of breath. The model is designed to render these as sounds rather than read them aloud. The model card recommends one or two tags per sentence.
Separate, short acting directions adjust emotion, pace, pitch, and breathing. This resembles giving actors performance notes alongside a script. Directions are optional; without them, performance follows the dialogue and tags.
3. A streaming engine for multiple environments
The accompanying engine unifies Mac, NVIDIA GPU, and CPU backends under one Python API, a command-line tool, and an OpenAI-compatible server. The server exposes POST /v1/audio/speech for use with existing OpenAI clients.
Audio streams out during generation in roughly 0.32-second chunks, while long inputs are split into sentences and streamed in order. The NVIDIA backend batches up to 8 concurrent requests.
Here is a simple Python usage example.
from lure_tts import Lure
tts = Lure.load("models/lure")
voice = tts.make_voice("reference.wav", "Transcript of the reference clip.", name="me")
tts.speak("Wait, you're staying? (sharp inhale) Since when?", voice,
instruct="Sudden, bright shock, then a caught breath.", out="out.wav")
for pcm in tts.stream("(sigh) Fine. I'm in.", "me"): # float32 chunks at 24 kHz
...4. H100 performance and Turbo speed
The repository measured single requests on an H100 after warm-up, using the default decoding context of 72 frames. Time to first audio (TTFA) depends on how many frames are gathered in the first chunk, a piece of audio sent at once. Batch slots are spaces for requests; both slot configurations were used in single-request tests.
Table: Streaming performance measured on H100
Conditions: H100, after warm-up, default decoding context of 72 frames, one request at a time
Units: TTFA in seconds, time per frame in milliseconds
| Model · batch slots | TTFA with 4-frame first chunk | TTFA with 1-frame first chunk | Time per frame after the first frame | RTF |
|---|---|---|---|---|
| Lure·1 | 0.283 | 0.229 | 18.4 | 0.23 |
| Lure·8 | 0.331 | 0.266 | 19.6 | 0.25 |
| Turbo·1 | 0.168 | 0.149 | 11.4 | 0.14 |
| Turbo·8 | 0.200 | 0.167 | 12.4 | 0.15 |
RTF is the ratio of generation time to audio duration; below 1 means generation is faster than playback. In the H100 table, Turbo has both lower first-audio latency and shorter per-frame processing time at the same slot count.
Turbo is a variant that reduces sequential audio code predictions per frame from 15 to 3. Audio codes are encoded values representing sound information. The repository reports about a 3.4x speedup in the code prediction stage when batching 8 streams on an L4.
In short-sentence measurements on separate hardware, an M5 16GB achieved an RTF of 0.79 with the MLX backend, while a CPU build on 16 vCPUs recorded 6.64. A vCPU is a virtually allocated processing unit. The former keeps up with real-time playback; the latter is suited to offline generation.
The repository tested 13 cases, generating each twice with the same voice and seed. Speaker similarity (ECAPA) on M5 was 0.533 for MLX and 0.527 for cpp Metal, below the reference run's 0.575 (qwen-tts, L4) and the 0.545 pass threshold. MLX achieves real-time speed but falls short on speaker preservation; the cause of the similarity drop remains unconfirmed.
5. Development considerations
Short recordings can define both voice and expression, but preparing inputs and checking results takes work. Choose a clean reference recording with one speaker and match the transcript exactly to the speech. Output also inherits the reference clip's microphone characteristics and room reverberation. The model card advises obtaining consent when cloning real people's voices.
If a rare sound tag is omitted, the recommendation is to regenerate with a different seed, the initial value for generation. Its ability to change accent, age, or gender through instructions is weaker than that of large instruction-tuned models.
Lure TTS's practical shift is to combine voice cloning and in-dialogue sound effects in a single input flow, then deliver the results through streaming.