cuda-bindings 13.3.0: Python DLA inference, GPU–DLA collaboration

1. Runtime management comes to Python

The new bindings are cuda.bindings.cudla in the CUDA Python project's cuda-bindings 13.3.0. Bindings expose an existing API to another language, here enabling device creation through inference task submission in Python.

The DLA runtime, the software supporting program execution, is included in JetPack and needs no separate installation. It runs models already compiled into executable code; TensorRT is needed only to convert models into that format.

The bindings require at least Python 3.10. NVIDIA also explains how to set up a 3.12 virtual environment for installation when needed.

2. Separate model conversion from on-device execution

A DLA loadable is a binary file that converts a trained network into a form the dedicated accelerator can read directly. Once built with TensorRT's trtexec, it requires no model conversion tool to run again for each inference.

NVIDIA's build example uses JetPack 7.2.2 or later on Jetson AGX Orin.

The tool versions are TensorRT 11.4.1 and CUDA 13.4 Update 1.

The command below targets an ONNX model with fixed input dimensions and 16-bit floating-point tensors. It selects the first DLA engine, the unit that runs inference operations, and specifies the input and output data layouts.

trtexec \
    --onnx=your_model_fp16.onnx \
    --useDLACore=0 \
    --inputIOFormats=chw16 \
    --outputIOFormats=chw16 \
    --memPoolSize=dlaSRAM:1 \
    --saveEngine=model.bin \
    --buildDLAStandalone

This command creates a DLA binary that runs without model conversion tools at inference time. Separating the build artifact from execution lets the Python application focus on loading the prepared model and submitting tasks.

3. Connect GPU and DLA through one task queue

Hybrid mode, where GPU and DLA work together, uses a CUDA stream, a queue for ordered tasks. Placing GPU preprocessing, DLA inference, and GPU postprocessing in the same queue links input preparation to result processing.

Execution starts by initializing CUDA, loading the model, and registering GPU memory with DLA based on the model's tensor information. Tensors are multidimensional numeric data. Inputs are copied and inference tasks submitted. Completed results are then copied to CPU memory for Python postprocessing.

Tensor information read from the model includes buffer sizes and data types; buffers temporarily hold data in memory. Using it for memory allocation and I/O layout reduces hard-coded model sizes. Returned byte arrays are also interpreted using this information for further Python processing.

4. Free up the GPU while managing models and memory

According to NVIDIA, offloading supported operations to DLA can improve power efficiency and leave GPU capacity for other application tasks. Developers can split work across accelerators while keeping coordination code in Python.

DLA supports fewer layers than the GPU, so operations around the model graph, the connected structure of operations, may need adjustment. Developers must handle buffer registration, copying, synchronization of task completion, and resource release. These bindings support only hybrid mode, and Python garbage collection does not release device handles.

5. Include error handling and shutdown in the Python flow

A failed cuDLA call raises a Python exception containing a status code and a readable error name. Developers can catch it and handle invalid arguments or hardware errors accordingly.

NVIDIA recommends placing cleanup code in try/finally. Registered memory, model modules and devices, GPU memory, and streams are cleaned up with matching release calls. This brings execution, error handling, and shutdown into one program flow.

cuDLA Python bindings let edge developers manage execution in one language when offloading inference to a dedicated accelerator and using the GPU for other tasks.