NVIDIA has released TensorRT 11.5 to optimize GPU inference for AI models. ARM servers can reuse a built engine across GPU generations, while GB300 support on Windows-based ARM devices expands developers' hardware choices.
1. CUDA 13.5 becomes the default development environment
TensorRT's default CUDA version is now 13.5. This changes build settings for developers using CUDA for GPU execution and TensorRT for model deployment. GitHub notes the change; official notes direct readers to platform support tables.
Building plugins requires C++20-ready sources. PLUGIN_VALIDATE, a macro expanded into code during compilation, no longer has its C++17 fallback. Beyond running built engines, environments for compiling custom extensions need review.
2. GB300 support expands to Windows-based ARM devices
GB300 support on Windows on ARM has been added in beta. Supported precisions match x86-64 environments with the same GPU compute capability. Model number formats range from FP32 to low-bit formats such as FP8 and FP4.
The change covers GB300's GPU execution path. GB300 has no separate deep learning accelerator, or DLA. Developers can focus adoption reviews on GPU and build tool support rather than constraints of other ARM devices with DLA.
3. One engine can serve more Linux ARM servers
Hardware compatibility now extends to Linux systems following the SBSA ARM server standard. An engine is a model prepared for GPU execution. Rather than build for each GPU, teams can deploy one engine across compatible generations.
Set HardwareCompatibilityLevel::kAMPERE_PLUS at build time to run the engine on supported Ampere or newer GPUs. NVIDIA cites GH200, Grace Blackwell systems, and DGX Spark. This can reduce engine creation and management for ARM teams.
4. Weigh deployment savings against runtime resource costs
Assess build migration, engine reuse, and model-specific resource increases separately. Check CUDA and C++ settings for source builds. For multi-generation deployments, check platform compatibility support before assessing reuse. Separately prioritize time and memory checks for models meeting the conditions below. Do not interpret increases under different conditions as a general slowdown or the cost of hardware compatibility.
ONNX is a standard model interchange format; strong typing explicitly sets tensor data types. NVIDIA reports the following possible resource increases for strongly typed ONNX networks on DGX Spark. An execution context holds inference execution state.
Table: DGX Spark resource changes relative to TensorRT 11.4.1
Conditions: DGX Spark, strongly typed ONNX networks; batch conditions not specified in the source
Unit: %
| Metric | Increase relative to TensorRT 11.4.1 |
|---|---|
| GPU processing time | About 10~20% |
| Execution context device memory | Up to 76% |
According to the explanation, this does not apply to GPUs with dedicated graphics memory.
NVIDIA's B300 and GB300 measurements cover a strongly typed FLUX.1-dev single-transformer attention network. Attention combines information based on context. At batch 4, meaning 4 inputs processed at once, execution context device memory may increase as shown below.
Table: B300 and GB300 execution context device memory by TensorRT version
Conditions: B300 and GB300, strongly typed FLUX.1-dev single-transformer attention network, batch 4
Units: memory in MiB, increase in %
| TensorRT 11.4.0 | TensorRT 11.5.0 | Increase relative to 11.4.0 |
|---|---|---|
| 1296 | 1728 | About 33% |
NVIDIA said it plans to address this issue in a future release.
5. Align build paths and ARM platforms first
Source builds now use CMake's find_package(CUDAToolkit) to locate CUDA. For a CUDA installation outside the default path, specify its location with the following setting:
-DCUDAToolkit_ROOTBuild configurations using -DCUDA_TOOLKIT_ROOT_DIR need to migrate to the new name.
However, this hardware compatibility feature is not supported on DriveOS or JetPack.
TensorRT version selection is separate. JetPack is not supported in this release. Jetson AGX Orin and Thor must use 11.4.1, while Jetson Orin on JetPack 6.x must use the platform-supported 10.x version.
On Windows on ARM, trtexec does not automatically register standard plugins, so parsing models that use them may fail. The source recommends building engines with Polygraphy as a workaround.
TensorRT 11.5 broadens ARM deployment options; adoption should prioritize source-build environment migration, cross-generation engine reuse, and resource increases for the stated models and hardware.
