Kapitsa, a local inference engine that computes answers from inputs, provides KFlash by default to speed up generation on Apple silicon. Developers using coding tools on Macs can check a small model's candidates as a tree rather than a line. This lets a large model generate more tokens in one computation.
Developers using coding tools or local models can make more use of candidates in the same computation pass. The project's published M4 Pro measurements showed 3.2~6.0% higher throughput for 512-token coding responses across three models.
1. Checking alternative candidates too
The block drafter, a small model that prepares token candidates, predicts candidates for the next 7 positions at once. The large model adds the last generated token and verifies 8 rows. But a conventional linear candidate chain discards rows after the first incorrect token. DFlash and DFlash2 rank 16 candidates per position by likelihood. Even if the first choice is wrong, the candidate the large model wants may still be on the list.
Like checking a second option at a fork, KFlash uses candidates and scores to build a branching tree, starting with the most likely branches. In the same 8-row computation, the large model follows the longest branch matching the tokens it generates. In the documentation's illustrative diagram, the chain emits 2 tokens and the tree emits 4, including the token added directly by the large model.
The candidate-generating model does not need retraining. The key change is a different use of the candidate lists and scores already provided by the released drafters.
2. Choosing verification sizes based on device costs
A budget mechanism sets the tree size by weighing expected token output and GPU time for each verification row count, choosing a size that favors throughput. It calibrates drafter confidence using records of accepted candidates and measures computation costs on each Mac.
In the project's measurements, Qwen3.6-35B-A3B selected an average of 4.8 rows for prose and about 6~7 rows for code and agent responses. Across 60 tree runs in the comparison experiment, predicted tokens per cycle differed from actual results by an average of 2.6%.
Apple GPU matrix computations use groups of 8 rows. Dense models, which compute with all parameters each time, share weight reads, so verifying 1 row and 8 rows costs about the same. This puts computation already being paid for to work on other candidates.
3. Speed changes observed in coding responses
According to the project, all three models delivered faster 512-token coding responses on a MacBook Pro M4 Pro 48GB. Tests ran on macOS 27.0.1, plugged in with High Power Mode and an exclusive GPU lock. The pre-KFlash and KFlash builds alternated, with five runs each.
Table: Throughput for 512-token coding responses
Conditions: MacBook Pro M4 Pro 48GB, macOS 27.0.1, High Power Mode, plugged in, exclusive GPU lock, 512-token coding responses, highest-probability token each time, 5 alternating runs per build, project measurements
Unit: tokens/second
| Model | Previous build | KFlash | Change (%) |
|---|---|---|---|
| Qwen3.5-4B | 179.3 | 188.5 | +5.1 |
| Qwen3.8-27B | 61.0 | 64.7 | +6.0 |
| Qwen3.6-35B-A3B | 176.1 | 181.7 | +3.2 |
For short responses, the gains from generating more tokens at once outweighed the extra computation costs. GPU time per cycle rose by 1~6% for dense models, which use the entire model. For the 35B mixture-of-experts model, which selects only some experts, it rose by 6~12% as each row called on different experts.
In long agent conversations, chains already emitted 2.9~6.4 tokens per cycle, limiting KFlash's throughput change to about −5%~+3%. Throughput is the amount processed in a given time. Some paths that handle multiple requests together verify chains rather than per-request trees.
4. Enabled by default, with a switch to the previous method
KFlash is enabled by default for models that use a block drafter. Users can start a local server with the following command.
kapitsa serve # chat + OpenAI/Anthropic APIs at http://127.0.0.1:8000To compare with the previous method, set the environment variable NI_KFLASH=0 to restore chain verification. After each request, server logs record the selected tree size, expected and actual token output, and costs, letting users inspect behavior on their own Macs.
KFlash's benefit is throughput gained through candidate exploration, and its extent depends on response length, model architecture, and verification costs.