The new release of llama.cpp, which runs language models locally, fixes duplicate copies of Gemma token embedding weights in CPU memory. These learned weights turn tokens into numerical vectors. Users who disable memory mapping, which maps model files directly into memory, can load the same model with less memory under the affected conditions.
1. Sharing input and output weights
The fix covers Gemma 3n and Gemma 4. Both reuse their input token embedding weights, learned values that turn tokens into numerical vectors, in the final output layer that produces the model's answer. Input and output effectively consult the same table, so two copies are unnecessary if they can share the same memory space.
Previously, the output layer was created first and could not find the embeddings to share. The fix creates the embeddings first and makes the output layer reference them. Rather than changing the weights themselves, it prepares the shared resource first to eliminate the cause of unnecessary copies.
The author explained that 94 other model architectures with shared input and output weights already used this order, with only these two models as exceptions. The fix brings their weight preparation order into line with the existing architectures.
2. Duplication occurred in CPU runs without memory mapping
The issue depends on both file loading and device placement. Duplication occurred when memory mapping was disabled and input and output used the same type of memory buffer. This includes CPU-only runs and runs that offload some layers to another device but keep the output layer on the CPU.
With default memory mapping, input and output weights point to the same mapped region, so no separate copy was allocated in the first place. Users checking the update's effect therefore need to examine how the model is loaded as well as whether it uses the CPU.
3. Buffer reduced by 216 MiB in the test model
The author loaded gemma-4-E2B-it Q4_K_M on the CPU alone without memory mapping. The CPU model buffer, which holds model weights, fell from 3163.73 MiB to 2947.73 MiB. The change proposal states that the 216 MiB difference exactly matches the size of one set of token embeddings.
Table: Gemma 4 CPU model buffer measurements
Conditions: gemma-4-E2B-it Q4_K_M, CPU only, no memory mapping, weight repacking disabled
Unit: MiB
| Item | Value |
|---|---|
| Before | 3163.73 |
| After | 2947.73 |
| Reduction | 216 |
The baseline is cf24317be, the development version before the fix. The reduction shown in the table is the difference reported by the author, representing memory freed by removing one duplicate embedding copy.
The author also checked the same model's CPU-only output using a method that selects the most likely token at each step. Output matched the pre-fix version both with and without memory mapping.
4. Reproduction command and scope of memory savings
Below is the exact test command from the change proposal. It uses CPU-only execution, disables memory mapping and turns off weight repacking, which rearranges weights for execution. It is used to check the buffer size reported during model loading.
llama-server -m gemma-4-E2B-it-Q4_K_M.gguf -dev none -ngl 0 -lm none --no-repack -lv 4The practical benefit is more room to fit a model when CPU memory is tight. Configurations that place the output layer on a GPU still incur the memory cost of a separate embedding copy on that device.
The published savings apply to the model and settings above; no separate execution speed measurements were provided.
b12008 connects Gemma's shared weights in the correct order, reducing duplicate memory in CPU runs without memory mapping.