ExecuTorch: Direct deployment to Cortex-M CPUs, device-specific ahead-of-time optimization

1. Prepare within the same tools instead of converting models

Previous Arm support targeted Ethos-U neural accelerators; this addition provides a Cortex-M CPU execution path. Avoiding a different framework does not mean skipping device-specific export or quantization, which represents values as small integers.

Before execution, the backend converts model operations into calls to CMSIS-NN, a collection of optimized operators for Cortex-M. It targets cores ranging from basic instruction sets to DSP and vector support. Core-specific operator selection is completed at compile time.

2. Shift runtime costs to advance preparation

Graph optimization, weight placement, and memory planning also happen ahead of time. The graph is the structure connecting operations. Preassigned locations for tensors and temporary buffers eliminate dynamic tensor memory allocation during inference. Tensors are the multidimensional numerical arrays used in model computation. This reduces runtime overhead, but developers must know the target CPU, instruction support, and memory capacity. They may also need to specify which data to place in fast memory.

Deployment proceeds through export, quantization using calibration data, re-export, conversion to device operators, and file saving. A dedicated quantizer creates an 8-bit integer graph meeting CMSIS-NN layout requirements. The following is the export portion of the original example.

# 1. Export
model = MyModel().eval()
example_inputs = (torch.randn(1, 1, 49, 10),)
exported = torch.export.export(model, example_inputs)

Packaging the resulting file into firmware with the runtime and CMSIS-NN library lets the device execute the pre-prepared graph.

3. Continue executing operations the accelerator cannot handle

On devices with Ethos-U, supported operations go to the accelerator while the CPU handles the rest within the same graph. CPU operations use CMSIS-NN when quantization, data layout, and shape requirements are met. Otherwise, they use unaccelerated general-purpose kernels, the implementation code that actually executes operations. Deployment remains possible even when some operations fall outside accelerator support, but those portions cannot be assumed to be accelerated.

4. Compare latency and memory under the same conditions

The authors measured four of the five models in MLPerf Tiny, an inference benchmark for small devices. The omitted fifth test, streaming wake-word detection, requires a test setup for inference across multiple frames. These single-inference figures therefore do not validate continuous-input workloads. The table below shows per-model latency and memory requirements under identical hardware and quantization conditions.

Table: Inference latency and memory usage for four models. Flash code is the code size with only required operators included; total flash also includes the model.

Conditions: Alif Ensemble E8, single high-performance Cortex-M55 core at 400 MHz, no Ethos-U NPU, INT8 quantization with CortexMQuantizer, Alif ML Embedded Evaluation Kit. Only required operators were built, using GCC 15.2.1, -Oz for the core runtime and -O3 for kernels, with logging disabled. These are internal measurements, not an official submission, so they cannot be compared directly with published MLCommons rankings.

Units: Latency in ms, memory in KiB

Under these conditions, all four models used less than 90 KiB of RAM.

5. Memory placement also affects performance on real devices

Conditions: Person detection demo, Renesas EK-RA8D1, Cortex-M85 at 480 MHz, INT8 µYOLO, 128×128 RGB input

Peak RAM was 254 KiB, and total flash was 640 KiB. Placing weights for costly operations in DTCM, data memory tightly coupled to the CPU, and enabling the instruction cache reduced inference latency from 191 ms to 24 ms.

The demo's overall throughput of 12.7 frames per second must be distinguished from inference time.

Future work includes broader mixed-precision coverage, which uses different bit widths for computations or weights. Connecting floating-point kernels, supporting more operators, and validating more hardware were also listed.

The value of this change lies in reducing conversion tools, while assessing actual deployment performance requires examining both operator support and memory placement on the target device.