Huawei CSL has released MACaQ, which creates deployable 4-bit models without loading the entire original model onto a GPU. It processes each linear layer independently to reduce quantization time and required GPU memory.
1. Quantizing both weights and activations to 4 bits with MXFP4
The paper quantized both weights and activations, the values passed during computation, to 4 bits, then compared quality, quantization cost, and execution speed separately. OCP MXFP4 is a 4-bit number format with shared block scales.
2. Improved Llama-3-8B quality on two datasets
The table below compares perplexity across MXFP4 W4A4 quantization methods, with unquantized FP16 as the baseline. Perplexity measures how well a model predicts what comes next; lower is better.
Table: Llama-3-8B MXFP4 W4A4 perplexity
Conditions: OCP MXFP4 W4A4, Llama-3-8B, WikiText2 training-set calibration, context length 2048, original values from Table 1. Calibration gathers statistics from representative inputs to reduce quantization error. FP16 is unquantized.
Unit: Perplexity
| Method | WikiText2 | C4 |
|---|---|---|
| FP16 | 6.14 | 9.46 |
| MR-GPTQ | 7.13 | 11.37 |
| DuQuant++ (GPTQ) | 7.17 | 11.66 |
| MACaQ | 6.90 | 11.01 |
Measurement notes: Experiments used one GPU with 504 TFLOPS FP16 compute, 96GB memory, and 1.79TB/s memory bandwidth. Evaluation used WikiText2 and C4 test sets after calibration. Defaults used 200,000 calibration tokens and 128 Flip-Refinement iterations, which reduce error by switching to neighboring 4-bit values.
In Table 1 of the paper, MACaQ achieved lower perplexity than MR-GPTQ and the DuQuant++ family, with the same ranking on C4.
3. Faster quantization, with serving gains at larger batches
The large Qwen3.5-122B-A10B model used 32.5GB of GPU memory for 64 minutes on one 96GB GPU. Only the expert layers of this MoE model, which selects experts for computation, were quantized. The paper and repository report 32 Flip-Refinement iterations and 512K calibration tokens.
Table: Quantization cost and quality
Conditions: Llama-3-8B, MXFP4 W4A4, original values from Table 7. One calibration block contains 2048 tokens; MACaQ's token stream is converted into equivalent blocks.
Units: Quantization time in seconds, perplexity
| Method | Flip-Refinement iterations | Calibration blocks | Quantization time (seconds) | WikiText2 perplexity |
|---|---|---|---|---|
| MACaQ | 0 | 1 | 14.8 | 7.11 |
| MACaQ | 128 | 31.3 | 88.1 | 6.91 |
| MACaQ | 512 | 488.3 | 431.0 | 6.86 |
| MR-GPTQ | – | 128 | 319.1 | 7.11 |
| DuQuant++ (GPTQ) | – | 128 | 624.4 | 7.16 |
In the table, reaching 7.11 perplexity took MACaQ 14.8 seconds and MR-GPTQ 319.1 seconds. The MACaQ rows increase both Flip-Refinement iterations and calibration blocks. Separately, the authors report matching the best quality of MR-GPTQ and DuQuant++ in 11x less time across the full cost-quality curve.
Speed varies by batch size. Marlin W4A16 is a matrix multiplication kernel that computes with 4-bit weights and 16-bit activations.
Table: Qwen3-32B decoding throughput by batch size
Conditions: Qwen3-32B, vLLM graph mode, 512-token prompt, median decoding tok/s, original values from Table 6.
Unit: tok/s
| Batch | BF16 | Marlin W4A16 | MACaQ |
|---|---|---|---|
| 1 | 22.8 | 71.8 | 69.8 |
| 64 | 1234.7 | 2697.8 | 2709.8 |
| 128 | 2086.1 | 3170.0 | 3983.3 |
| 256 | 3061.5 | 3573.3 | 5252.8 |
The table shows similar performance for MACaQ and Marlin at batch 1. As batches grow, Marlin's gains diminish while MACaQ keeps rising, opening a gap from batch 128. The paper says MACaQ matches Marlin when memory reads are the bottleneck and leads when computation is the bottleneck. This is a reason to compare at the target batch size for serving grouped requests rather than single requests.
4. Streaming quantization through within-layer transforms
MACaQ applies per-channel scaling and block Hadamard rotation to activations, using orthogonal rotation to spread the distribution more evenly. It absorbs the inverse transform into the weights. Since transforms do not propagate beyond a layer, there are no cross-layer dependencies, enabling streaming with only one layer in memory at a time. The authors note that this gives up cross-layer error compensation.
Block Hadamard rotation reduces losses from 4-bit representation. However, it must run during inference, adding cost. For Qwen3-32B vLLM decoding at batch 1 and context 512, the paper reports 2.3% overhead over plain MXFP4 and says the proportion decreases as models grow.
Flip-Refinement uses the Hessian, a matrix representing the input distribution, to reduce error by moving each FP4 code to a neighboring representable value. In the paper's ablation table, Llama-3-8B perplexity fell from 7.08 with per-channel scaling to 6.87 after 512 iterations, using 200,000 calibration tokens.
5. Extending to GGUF correction and image generation
Consider MACaQ-R if you have an MoE model already quantized in the GGUF storage format, along with its matching original model, teacher model, and calibration data. KL measures prediction-distribution divergence from the original model; lower is better. For Qwen3.6-35B-A3B, the repository table lists MACaQ-R Q4_K_XL at 22.36GB with KL 0.013367, versus Unsloth UD-Q4_K_XL at 22.36GB with KL 0.015731.
If standard llama.cpp is installed and the MACaQ-R GGUF is saved at the path below, you can run it with this command from the README.
llama cli -m models/model-macaq-r.ggufTable 3 evaluates W4A4 quantization of only the transformer portion (DiT-only) of the FLUX.2-klein-4B image generator at 1024² and 6 steps, using 800 GenEval2 prompts. Against BF16 outputs with matching prompts and seeds, MACaQ had higher PSNR, a pixel-level similarity metric, than official NVFP4: 15.04 versus 14.85. LPIPS, a perceptual difference metric, was lower: 0.218 versus 0.235.
These figures come from different model, runtime, and GPU combinations, so results from one setup should not be used as evidence for another.
Quantization preparation cost and actual serving performance are separate dimensions, so the key is to choose a path suited to the target batch size and runtime.