FlashInfer, which helps run GPU inference computations for AI models, released a new version on October 9, 2026. More ways to distribute model computations across GPUs give developers building distributed inference on Rubin or Blackwell workstations additional execution paths to consider.
1. Expanded execution paths, not first-time support
Mixture of experts (MoE) selects a subset of a model's expert computations for each input. It is like assigning work only to the team members needed. Expert parallelism places these experts across GPUs. Quantization reduces the number of bits used to represent numbers.
W4A16 computation for workstations already existed in v0.6.14. It uses 4-bit weights and 16-bit activations for intermediate computations.
The preceding v0.7.0 also had an MXFP8 expert-parallel path for workstations. This update expands supported formats and execution paths rather than introducing distributed quantized experts for the first time.
2. Check supported formats and combination methods on Rubin
Rubin gains distributed execution paths for NVFP4 and MXFP8 experts. Expert results are combined and final outputs produced in a 16-bit format.
The integration entry point to check is MoEEpLayer, with BF16 used for combination and output. The change proposal says in-kernel summation is disabled by default because multiple operations may update the same value in varying orders.
3. On workstations, the caller handles communication
The Blackwell workstation targets are SM120 and SM121. This path adds distributed experts to the previously described computation using 4-bit weights and 16-bit activations.
The optional expert_map argument specifies expert placement per GPU. Each GPU receives the full input batch and computes only its own experts. Execution processes on different GPUs do not exchange tokens. Tokens are pieces of text processed by the model.
The benefit is that expert computations can be split across cards. The calling application must still deliver the full input to each GPU and combine partial outputs. Integration costs must therefore include input delivery and output summation, not just expert computation.
Support restrictions also vary by implementation. The separate B12x NVFP4 implementation in this change proposal still rejects expert parallelism because it locates each expert's storage using model-wide expert indices.
The new CuTe DSL-based NVFP4 implementations introduced separately in the release share a prepared expert weight layout. A single weight copy allows 4-bit activations for initial input-context processing and 16-bit activations for token-by-token generation. Expert-parallel support for these new implementations was not specified.
4. The 1.36× figure compares computation in one layer
The performance measurement used DGX Spark and the Qwen3.8-Flash-Next model. A single MoE layer of this model was run with one token.
The release says the W4A16 implementation is 1.36× faster than Marlin under these conditions. The full measurement method and expert-parallel configuration were not provided. This does not justify treating the figure as a speedup for Rubin or an entire multi-GPU service.
Correctness checks are separate from speed comparisons. The workstation change proposal reported that results from two RTX Pro 6000 GPUs matched those from a single GPU running all experts.
The Rubin change proposal also reported passing single-GPU and four-process expert-parallel checks in a separate code version using the same execution code. Neither change proposal provided numerical speed results.
The value of this expansion lies in more placement choices for quantized experts, and adoption should be judged by the chosen implementation's support scope and the full execution path.