ExecuTorch 1.3.1: Streaming speech recognition latency analysis, Arm SME2 optimization

Results

Original total processing times. QC8W uses dynamic INT8 activations and per-channel INT8 weights.

Conditions: Google Pixel 11 · 24.9-second recording · 1,120 ms chunks · 1 inference thread (pinned to a middle core) · ExecuTorch 1.3.1 SME2 on/off · 2 warm-up runs + 1 measured run

Unit: processing time in seconds

The real-time factor is processing time divided by audio duration. The authors say both configurations run faster than real time, leaving room for other CPU tasks. However, the first text must wait for audio collection, preprocessing, and inference.

Why the difference?

SME2 is Arm's matrix computation extension. In the authors' ETDump analysis, matrix multiplication (GEMM) accounted for the largest share of time. Enabling SME2 reduced both its time and share.

Table: GEMM execution time comparison from ETDump

Conditions: ETDump operator time · Pixel 11 · SME2 on/off comparison

Unit: seconds

In the same analysis, GEMM's share fell from 84%→67% for FP16. For QC8W, it fell from 78%→62%. Data movement time, such as copying and concatenation, barely changed and remained a relative bottleneck. This work limits overall speedup and is a target for further optimization.

The authors' Arm Streamline measurements showed SME utilization of 38.4% for FP16. For QC8W it was 53.7%. This is the proportion of matrix hardware allocation, not a percentage of peak compute speed.

The authors' kernel sampling confirmed execution of KleidiAI's SME2 GEMM kernels. Some GEMMs still used the NEON vector path in the FP16 build and the FP32 path in the QC8W build.

Selection criteria

Original accuracy results by chunk size. WER stands for word error rate.

Conditions: host measurements · ExecuTorch runtime + XNNPACK CPU backend · 14,019 utterances across 4 corpora: LibriSpeech Other, AMI Cleaned, Earnings22, and VoxPopuli Cleaned AA

Unit: WER in %

Longer chunks lower error rates but delay the first character. The authors' estimated first-output latency for 560 ms chunks with SME2 enabled is 691 ms for FP16. For QC8W it is 660 ms. They added audio collection time to the median execution time of the model's three processing stages. Measurements used synthetic input and initialized stream states over 3 rounds, with 75 samples per configuration. The estimate assumes the first output is text.

Applying the case study's 800 ms latency budget and 13% WER ceiling together makes 320 and 560 ms configurations candidates. The 80 ms option exceeds the error target, while 1,120 ms exceeds the latency target. The 320 ms option favors fast responses, while 560 ms favors accuracy; at 560 ms, QC8W offers more latency headroom. Use QC8W to save time when accuracy headroom is ample, and FP16 when it is tight. In the case study, FP16 differed from FP32 by less than 0.03 percentage points. QC8W added 0.21~0.31 percentage points of WER over FP32.

Measurement method

For reproduction, keep the Nemotron 3.5 model artifacts identical and retain XNNPACK and KleidiAI, toggling only SME2. Device conditions are those in the first table.

ETDump shows operator times, while Arm Streamline shows native function samples.

The required CMake settings in the ETDump documentation are:

-DEXECUTORCH_BUILD_DEVTOOLS=ON
-DEXECUTORCH_ENABLE_EVENT_TRACER=ON

With ExecuTorch 1.3.1, verify the SME2 build and analyze ETDump data with the Inspector API, while treating these figures as results from one Pixel 11 and one Nemotron 3.5 model.