NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB: Compression through 2-bit training, local vision-language agents

1. What runs locally

Released on October 15, 2026, Bolt accepts text, images, and video and outputs text.

It uses a mixture-of-experts architecture with 74 billion total parameters, activating 9 billion for computation. Rather than computing every part each time, it selects the parts it needs.

Its main use is local agents that connect web browsing, terminal use, and tool calls. When the model requests a tool call, the application executes it and returns the result, so deploying the model alone does not complete task automation.

It is available for commercial and noncommercial use under the OpenMDW-1.1 license.

2. Compressing a large model instead of choosing a small one

A larger 120 billion-parameter vision-language model was reduced using Star Elastic. This compression method selects expert counts and internal widths per layer to fit a memory budget. Quantization-aware distillation then corrected low-precision errors during training. Expert weights are stored in IQ2_XXS at 2.0625 bits per element, with higher precision retained in some layers.

On the tested hardware, the bandwidth for reading weights from memory is the main speed constraint. Structural compression and low-precision storage reduce weight memory usage and the data to read, helping accelerate inference.

The benefit is fitting a large model's capabilities into limited memory. The cost went beyond a simple file conversion: additional training was needed to restore visual understanding and agent capabilities after compression. NVIDIA says this was better than training a smaller model from scratch and reducing it to 4-bit precision under the same memory budget.

The model card reports terminal task scores of 50.98 for Bolt and 44.38 for Qwen3.6. However, Bolt trails Qwen on some knowledge accuracy and hallucination suppression measures, so this agent evaluation advantage cannot be generalized to overall accuracy.

3. Check setup requirements before running examples

The following example prepares the container and kernels for the minimum vLLM version specified in the model card. The full server launch command is in the original quick-start section; the code below does not replace it.

docker run -it --name vllm --gpus all --ipc=host -p 8000:8000 \
  --entrypoint /bin/bash \
  vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989

pip install b12x
vllm download-kernels

The 24GB model size must be distinguished from the memory needed to run it. The maximum context is 256K tokens. Questions, images, conversation history, tool results, and generated answers share this space. At deployment, it can be reduced to suit available memory and concurrent request counts. The architecture search targeting the memory budget used 1 request and a 128K context, so it does not guarantee the maximum context within 24GB.

Thinking can be enabled or disabled. The thinking process and final answer share the same output token budget, so check that budget if an answer is cut off by a length limit.

4. Read speed figures alongside test conditions

The technical blog measured all models on the same hardware with 128GB of memory.

Coding throughput was measured with controlled input lengths in AIPerf SPEED-Bench-Coding. AA-AgentPerf Local measured replay time across multiple complete agent task trajectories. Higher is better for the former; shorter is better for the latter.

With IQ2_XXS quantization and a LiLiCorr draft model, Bolt achieved coding throughput of 83.99 tokens per second and a task trajectory replay time of 600.80 seconds. A draft model accelerates inference by proposing candidates for the next token in advance.

The representative baseline, Qwen3.6, recorded 96.63 tokens per second and 453.97 seconds on the same two metrics. Gemma 4 recorded 100.56 tokens per second and 480.68 seconds. Under these hardware and execution conditions, Bolt had lower throughput and longer replay times than the two smaller mixture-of-experts models.

However, it was faster than Qwen3.8, which recorded 28.26 tokens per second and 1043.28 seconds.

Comparison conditions: The accuracy evaluation is Terminal-Bench 2.1. It compares NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (IQ2) with Qwen3.6-35B-A3B in BF16, a 16-bit numerical format. This accuracy comparison does not use the same conditions as the low-precision speed tests.

Execution conditions: Speed was measured on RTX Spark (Surface Laptop Ultra, 128GB) using llama.cpp and Windows 11 ARM x64. The comparison models, Qwen3.6 35B-A3B, Gemma 4 26B-A4B, and Qwen3.8 27B, all used UD-Q4_K_M quantization. The first two used MTP inference acceleration, while the last used DFlash inference acceleration. Because quantization and acceleration methods differ, these results should not be interpreted as a ranking of the models themselves.

5. Check support before deployment

The model card summary also lists deployment targets including A100 and H100-80GB.

However, the software integration section specifies Blackwell SM120 and SM121, and Linux. Since the stated support scopes differ, check the execution recipe for the actual target hardware.

Bolt outscored Qwen3.6 in a terminal task agent evaluation but was slower than two smaller mixture-of-experts models in separate speed tests, so selection should weigh both task evaluation results and response speed.