Developers fine-tuning large language models in their own environments now have a path that keeps part of the model compressed on the CPU during training. The new version trains only small added weights and supports connecting saved results to an inference server.

1. Training while keeping weights compressed

KTransformers v0.7.2 was released on October 15, 2026.

The supported model is the April edition of DeepSeek-V4-Flash. Routed experts, the compute modules selected for each input, run on the CPU, while other modules, including attention, run on the GPU.

LoRA freezes existing weights and trains small added correction weights. This path retains MXFP4, the experts’ original 4-bit compressed format, for prediction and training gradient computation. Crucially, it avoids keeping a permanent copy of the frozen expert weights expanded into a wider numerical format.

2. Training tool integration and costs

Integration with LLaMA-Factory supervised fine-tuning requires matching the validated integration package and dependency lockfile. The extra dependency bundle in the standard installation does not provide the required development branch.

The validated training setup uses ordinary conversations, including multiple turns, without reasoning generation mode. The training length is 1024 tokens.

Automatic conversion uses BF16, a 16-bit numerical format. Multi-GPU training uses FSDP2 to distribute training state.

The benefit is keeping expert weights compressed in storage. GPU intermediate results are recomputed, while CPU expert intermediate results are retained. The first distributed process handles the experts’ added weights and optimizer state. Non-expert weights are unpacked for training and stored in a separate cache, requiring about 13.7 GiB of disk space for the April edition.

The cache must reside in a dedicated directory separate from the source, which must remain accessible. It is created on the first run, then validated and reused on later runs.

3. From checkpoints to inference adapters

Keeping only the regular adapter, a correction weight file, omits the CPU experts’ added weights. Keep save_only_model: false and preserve the entire checkpoint directory containing training state. Resuming requires the same model, cache, distributed process count, training targets, and total learning-rate schedule duration.

The export tool below checks source identity, hashes, and added-weight shapes, among other things, without modifying the source.

python -m kt_kernel.sft.export_dsv4_sglang_adapter \
  --source /models/DeepSeek-V4-Flash \
  --cache /cache/v4-non-experts \
  --adapter /runs/v4/checkpoint-20 \
  --output /deploy/v4-adapter

The output consists of a model with regular added weights merged in and separate added weights for CPU experts. Both are required.

SGLang inference requires a matching dependency lockfile and keeps experts on the CPU. The validated request length is 2048 tokens for input and output combined.

4. What the measurements establish

Independent kernel validation before training integration used a single expert layer of DeepSeek-V4-Flash-0731. The server had two AMD EPYC 9355 processors, with 64 physical cores and 128 hardware threads.

The test used AVX512-BF16, not AMX. It used layer 3 of a real checkpoint with 256 experts, a hidden dimension of 4096, an intermediate dimension of 2048, 6 selections per token, an added-weight rank of 8, and a token length of 2.

Compressed layer storage was 3,623,878,656 bytes. Peak resident memory was 11.699 GiB for TP1 with 1 partition and 11.623 GiB for TP2 with 2 partitions. These figures include source tensors, temporary partition preparation space, and compressed kernel copies.

Results were compared with reference values from the expanded weights of the 6 experts selected by the first token. Relative L2 error, measuring value differences, was at most 0.007374. Cosine similarity, measuring directional alignment, was at least 0.999973.

Synthetic tests on the same machine used 4 experts, hidden and intermediate dimensions of 256 each, 2 selections per token, and an added-weight rank of 8. They used 1 or 2 partitions and 1, 2, or 8 training gradient accumulation steps. Against reference values in FP32, a 32-bit numerical format, relative L2 error was at most 0.008501 and cosine similarity was at least 0.999965.

5. Remaining limits for adoption decisions

In the real-layer test, the hash used to check whether the source remained identical did not change after the added weights were updated.

Observations from runs with a token length of 2 verify correctness, not throughput or latency. No full-model measurements of loss, quality, resumption, or throughput were provided.

The integration documentation also excludes packing multiple conversations into one training sequence, bulk targeting of all modules, dynamic adapter swapping, and other model variants from its supported scope.

This version’s value lies in connecting compressed expert training to saving and inference, while adoption requires planning package combinations and resource allocation together.