audio.cpp v0.9.1 was released on October 7 (UTC). Among many changes, it carries an optimization aimed narrowly at the PocketTTS CPU path.

The previous CPU Mimi upsampler expanded diagonal weights into a 32 MiB dense representation. The change uses the framework’s existing 64 KiB depthwise module and keeps streaming overlap, session-owned graph caches and the original GPU paths.

Only two PocketTTS source files change, with no framework, ggml or interface changes.

Why a depthwise module replaces the dense expansion

Diagonal weights act on each channel independently, so expanding them into a dense matrix means carrying a mostly zero representation. A depthwise (per-channel) module keeps that structure as it is.

This is a different kind of gain from shrinking the model or lowering output quality. A GPU deployment has no basis to expect a benefit from this CPU-only change.

In streaming speech, time to first audio and time to finish an utterance are different questions. A steady-state decoder number points to a local bottleneck but does not describe how fast a new session starts speaking.

Measurement scope of frame medians and session RTF

Real-time factor (RTF) is processing time divided by audio duration; lower is faster. With the same Debug build and request at four threads, decoder time per frame fell from 30.50 to 10.11 ms and session RTF from 0.461 to 0.207.

At two threads the figures were 34.98 to 14.58 ms and a session RTF of 0.557 to 0.302. Whole-process CPU peak RSS fell from 663.5 to 603.9 MiB.

Frame medians exclude the first two invocations, while the session figure includes graph initialization. Streaming and consecutive-request output stayed byte-identical on CPU, CUDA and Vulkan; Metal and HIP were not run.

All of these are maintainer measurements, not an independent reproduction.

A slower number on the HTDemucs CPU path

The release’s HTDemucs change shows why “faster everywhere” does not fit. On a warm request with an eight-second input and eight threads, CPU went from 5.056 to 5.407 seconds, about 7% slower.

Under the same conditions, Vulkan (RTX 5090) went from 248 to 148 ms, about 40% faster.

The old CPU implementation ignored the batch dimension HTDemucs uses for time frames. After the rewrite, drums/other cosine similarity against official Python rose from 0.683/0.924 to 0.999974/0.999998, so the timing compares paths of different correctness.

Checkpoints for remeasuring under matched conditions

For a deployment decision, matching the comparison conditions is the realistic approach. Fix the model, input text, thread count and output mode, and record fresh-session and repeated-request behavior separately.

Beyond elapsed time, continuity across streamed chunks and consecutive requests is also worth checking. These are checks for the target system, not tests performed for this article.

Whether the CPU supports the AVX-512 that release binaries may require can be checked first with standard Linux tooling.

grep -o -m1 avx512f /proc/cpuinfo

If the output is empty, the portable build shipped with the release (for example audio-v0.9.1-bin-ubuntu-x64-cpu-portable.tar.gz) is the relevant one.

If that CPU stage gets cheaper but another stage dominates request time, the application-level gain will be smaller than the local improvement. Avoiding a large temporary representation still appears worthwhile for memory on its own.

AVX-512 builds and portable builds

The release notes say recent builds tend to target AVX-512 because of automatic CPU detection on GitHub runners. For crashes, they point to the portable version for broader CPU compatibility.

A crash from an incompatible executable is not evidence that the model or the optimization itself is unsupported.

This patch is not a speed multiplier for the whole library but a fix to the cost structure of one path, the PocketTTS CPU upsampler.