llama.cpp Xing4.0: CPU and CUDA inference graphs, GGUF conversion

1. Support from model registration to inference

The pull request merged on November 5, 2026 includes architecture and hyperparameter registration, tensor loading, conversion scripts, and inference graphs. These define model structure, settings, arrays, and execution order. The changes were merged in 16 commits, with 28 checks passing at merge time.

The PR author reported verifying CPU inference on an Intel Xeon Platinum 8468 and CUDA inference on an NVIDIA H800 80GB. Both environments reportedly passed tests checking basic functionality.

2. Graphs built from existing operators

The inference graph combines existing components instead of adding new ggml operators. For mHC, which mixes residual connections from earlier layers into later ones, it also reuses fused operators that bundle computations.

MTP, which predicts additional tokens, was connected to the existing speculative decoding framework, which accelerates generation by verifying draft tokens.

3. GGUF conversion and MTP weight selection

The conversion script registers the new model in the Hugging Face-to-GGUF conversion path. It adapts existing code to correct positional encoding settings and uses a SentencePiece-based tokenizer to split text into tokens.

Conversion can omit the MTP block for additional predictions, or create a separate file containing only that block and shared embedding, normalization, and output weights. These map tokens to vectors and adjust value ranges. The original code below filters which weights to save for the two options.

        # --no-mtp: drop the appended NextN block entirely.
        if is_mtp and cls.no_mtp:
            return None
        # --mtp: keep ONLY NextN-block tensors plus the shared embeddings/
        # norm/lm_head (so the resulting GGUF carries just the draft head).
        if cls.mtp_only and not is_mtp and name not in (
            "model.embed_tokens.weight", "model.norm.weight", "lm_head.weight",
        ):
            return None

4. Quantization quality shown alongside baseline precision

The author reported perplexity by weight format for a 29B model. This metric measures how well text is predicted, with lower values being better. Results cover the bf16 baseline and two formats with reduced weight precision.

Table: Perplexity by weight format for the 29B model

Conditions: 29B model

Unit: PPL

In the quality table, q8_0 was close to the baseline precision, while iq4_nl was higher.

The author specified imatrix.dat to protect the precision of weights sensitive to quantization and said it would be released soon.

5. GPU throughput gains and additional computation costs

The author measured speed using llama-bench on an H800 80GB with CUDA. Input processing, pp512, and token generation, tg128, were reported separately. The error columns contain the values shown after ± in the original.

Table: H800 80GB CUDA throughput

Conditions: NVIDIA H800 80GB, CUDA, llama-bench pp512/tg128

Unit: Tokens per second

In these GPU measurements, both quantized formats outperformed baseline precision in input processing and token generation throughput. The original did not specify whether the quality and speed tables used the same model size.

Developers can now use the model on the existing execution infrastructure without new operators. However, scaling the post gate by a fixed 2.0, separately from learned hc_scale weights, adds computation nodes and GPU kernel launches per layer. The author noted that simply removing this scaling halves the post gate and changes the results, identifying it as a future optimization target.

The merge connects Xing4.0 weight conversion to CPU and CUDA inference through the existing execution infrastructure, leaving further computation optimization as follow-up work.