On November 13, Higurashi AI released weights, training statistics, and execution code for Haru-4B, a voice conversation model that listens and speaks simultaneously. Voice service developers can test backchanneling and overlapping speech in a web demo or their own programs. They can also specify speaking style and voice.
1. Listening and speaking simultaneously instead of waiting for a turn
Unlike turn-based conversations, where each speaker waits for the other to finish, this model is designed to speak while listening, as in a phone call. It receives user audio 7.5 times per second, about every 133 milliseconds, and generates model speech and text together.
Internally, user audio, model audio, and model text are merged into a single input position. According to the company, this cuts context length, the amount of information the model can consult at once, to one-third of that used by token-level interleaving of the three streams. Instead of a separate speech-generating transformer, a neural network architecture for computing context, a small neural network outputs audio in one generation step per frame. Each frame is a fixed-duration audio segment, and this approach reduces the number of sequential computation steps.
2. Learning conversational rhythm from real recordings
According to the company, pretraining, the process of learning from data before the main training stages, used over 40 million hours of real speech in more than 160 languages. It focused on 9 languages, including Korean. Dialogues created by language models and synthetic, computer-generated speech were excluded from training data. The focus was on learning hesitation, backchanneling, and overlapping speech from real conversations.
Training was split into adaptation to single-channel audio, recorded through one channel, and simultaneous voice conversation training. The company says it separated recordings of multiple speakers into distinct, time-aligned audio streams, obtaining over 13 million hours of dialogue data. Conversations up to 60 minutes long were used for training.
Learning natural conversational timing comes at the cost of large-scale data preprocessing, the preparation of data before training, and computing resources. According to the company, pretraining used 256 NVIDIA B200 GPUs for about 8 days. With the released weights, developers can begin experimenting with voice interaction without performing this training from scratch.
3. Running web demos and local apps with released weights
The weights and code were released under Apache License 2.0. The repository includes a PyTorch implementation with a web demo and a native app for Apple silicon that runs directly on the device. A vLLM-Omni integration plugin is also provided for handling multiple conversations on GPU servers.
The web demo requires an NVIDIA GPU with about 12GB of memory. Connect via a local address or HTTPS for browser microphone access, and supply the voice the model will use. Reference audio should contain clean speech from one person and last between 3 seconds and 3 minutes.
The example below shows the repository's Python usage instructions. It sets the voice using reference audio, starts a conversation session, and then passes user audio frame by frame.
from haru import DuplexSession, HaruModel
from haru.audio import load_audio, prepare_reference
model = HaruModel.from_pretrained("higurashi-ai/Haru-4B")
voice = prepare_reference(load_audio("voice.wav"))
session = DuplexSession(model, system_prompt=card, reference=voice)
frame = session.start() # the model's first 133 ms: frame.audio, frame.text
while not frame.done:
frame = session.step(user_audio) # 3200 samples at 24 kHz in, 3200 outThe model card notes that this research base model has limitations in factual reliability and instruction following at the level of a general-purpose assistant. Reference audio must use a voice you have the right to use.
4. MacBook latency is about 65 milliseconds including audio input and output
The company reports a median end-to-end latency, the time from input to output, of 65.2 milliseconds in a native app with custom Metal kernels on an Apple M1 Max MacBook. The measurement used int8 weights and a wired microphone and headphones, excluding the time needed to collect audio into frames.
Table: End-to-end latency measured on Apple M1 Max
Conditions: Apple M1 Max MacBook, native Swift app with custom Metal kernels, int8 weights, wired microphone and headphones, audio framing excluded (company-reported)
Unit: ms
| Item | Latency |
|---|---|
| Input (microphone→app) | 5.3 ± 0.0 |
| Model 1 step | 34.2 ± 0.5 |
| Socket and handoff | 4.1 ± 0.3 |
| Output wait (15 ms margin and remaining time) | 13.3 ± 1.5 |
| Output (app→headphones) | 8.1 ± 0.2 |
| Total (p50) | 65.2 ± 1.5 |
Model computation has the highest latency among individual stages, but input, output, and waiting time also contribute to total latency. The company says collecting audio into frames adds up to 133 milliseconds, which must also be considered when designing conversational responses.
The published GPU server figures cover computation rather than end-to-end latency. The company reports about 15 milliseconds for audio encoding, decoding, and one language model step on an NVIDIA B300. While the MacBook measurement shows local audio input and output, this figure shows the server computation time for processing one frame.
Haru-4B's key contribution is pairing released code with realistic conversational timing, giving developers a foundation for testing voice interaction themselves.
