llama.cpp b12144: Improved CUDA MoE tile layout, F16 performance regression fix

1. Rescheduling expert workloads by tile

For CUDA matrix multiplication in mixture-of-experts models, an auxiliary operation now preassigns small computation bundles called tiles. Matrix multiplication multiplies numbers arranged in rows and columns. Previously, only each expert's computation range was passed along. The new approach reduces the main operation's tile count and overhead while making work distribution among experts more flexible.

2. Throughput measured on a medium-sized mixture-of-experts model

According to the pull request (PR) author, the data layout change itself improves performance by 2–4%. The remaining gains come from reducing wasted computation with smaller tiles. The optimization favoring small tiles, previously limited to certain GPUs, now applies to a broader range of GPUs.

Below are some of the PR author's measurements for a medium-sized model. The model is gemma4 26B.A4B, with Q4_0 quantization, a process that reduces the numerical precision of the model.

The test processed an 8192-token prompt with a microbatch size of 64, the size of each processing chunk. The table compares the existing master with bea596f39, showing each GPU's change relative to its own baseline.

Table: Medium-sized mixture-of-experts model throughput by GPU

Conditions: gemma4 26B.A4B Q4_0, pp8192, microbatch 64, master versus bea596f39 (PR benchmark table)

Units: Throughput in tokens per second (t/s), speedup as a multiplier

With the same model and settings, the RX 6800 reached 1.47x its baseline throughput, while the MI60 / MI50 reached 1.43x. Some other listed devices showed little change, indicating that the optimization's impact varies by hardware.

3. Fixing the F16 regression by skipping empty work

A half-precision (F16) performance regression was reported during PR review. The proposed fix immediately ends computation for experts with no assigned columns. The code change is as follows.

    const int expert_end   = ts.col_high;
    const int ncols_expert = expert_end - expert_start;

    if (ncols_expert == 0) {
        return;
    }

    GGML_UNUSED(channel_ratio);

Below are measurements showing a regression on a small mixture-of-experts model, with throughput ratios below 1 relative to the baseline.

Table: Throughput regression examples before the F16 fix

Conditions: granitemoe 3B.A800M F16, pp8192, microbatch 32, master versus bea596f39 (PR benchmark table)

Units: Throughput in tokens per second (t/s), speedup as a multiplier

The fix to skip empty work was added after bea596f39, so this table shows results before the fix. The PR author said measurements on Granite 3.0 vary widely because whether the fixup kernel runs has a major impact on total time.

4. Weighing both benefits and costs

Small tiles reduce waste by assigning work more closely to the computation range. In exchange, precomputing and storing the new layout information moves into an auxiliary operation. The changed code also allocates temporary memory for per-tile information instead of per-expert computation ranges.

Verification tests on dense models, which compute without routing work among experts, showed little change. In the PR's llama 8B Q4_0 test with pp1024, throughput ratios relative to baseline were 0.99–1.03.

This improvement focuses on reducing wasted computation in mixture-of-experts models rather than broadly accelerating dense models. The b12144 release notes include both the CUDA changes and the half-precision regression fix.

b12144 improves CUDA processing for mixture-of-experts models and fixes an F16 regression, but practical performance should be assessed for each GPU and model setup.