Researchers at New York University and the University of Calgary proposed deletion tuning, which learns new capabilities simply by zeroing some large language model weights. Developers adapting models to tasks can store a small deletion list instead of saving all weights again. In the experiments, larger models approached the accuracy of training that changes all weights and better retained performance on prior tasks.
1. Learning where to delete instead of rewriting weights
Deletion tuning targets gate projection weights, the learned values in layers that regulate information flow within a model. Training produces a binary mask, an array specifying whether to keep each value or set it to 0. Rather than rewriting existing numbers, it marks positions to retain or delete.
During training, continuous values adjust deletion decisions before a final choice between retention and deletion. Initial choices use Wanda scores, which estimate weight importance from weight magnitudes and input activations. The deletion rate is not fixed. Across the six models in the main reinforcement learning comparison, less than 0.05% of all weights were deleted.
2. Large models approached full-weight training
Researchers evaluated with greedy decoding, which selects the most probable next token at each step. They reported checkpoints, or saved training states, with the highest validation accuracy. The comparison below trains the same model on math using reinforcement learning, which improves responses based on rewards. It also includes LoRA, a method that trains only additional low-rank matrices.
Table: Math accuracy and storage size for the same 8-billion-parameter model
Conditions: Llama3-8B-Instruct, GSM8K, GRPO, best validation checkpoint, greedy decoding, H200 GPU and FSDP. Storage is a lower bound on disk space needed to reconstruct the model using full fine-tuning fp32 weights or compressed masks.
Units: Accuracy %, storage size MB (1 GB = 1024 MB)
| Method | Accuracy | Storage size |
|---|---|---|
| Base model | 60.27 | 0 |
| Full fine-tuning | 83.24 | 30633 |
| LoRA(r=1) | 83.38 | 10.0 |
| Deletion tuning | 83.55 | 8.4 |
Under these conditions, deletion tuning achieved accuracy similar to full fine-tuning while reducing the changes that must be stored. The base model's storage size of 0 means no additional training results are stored beyond the original model.
A gap remained for smaller models. On the same math task, the 1-billion-parameter model scored 54.44% with deletion tuning and 60.50% with full fine-tuning. Qwen2.5-14B trained on a separate math dataset, SimpleRL-Zoo, averaged 55.43% and 54.87%, respectively, across six benchmarks.
On the same six benchmarks, Qwen3-30B-A3B scored 58.54% and 58.43%, respectively. This extends the method to a mixture-of-experts model that activates 3 billion of its 30 billion parameters, using only some expert layers for each input.
3. Also applied to answer imitation and self-distillation
The method also worked in supervised fine-tuning, which adjusts a model using examples of correct answers. On MMLU-Pro, a knowledge evaluation, the 3-billion-parameter Llama model averaged 22.28% with deletion tuning and 21.19% with full fine-tuning. Both methods had the same average accuracy on the reasoning evaluation BBEH.
Self-distillation transfers teacher predictions to a student by learning the teacher's next-token predictions on responses generated by the model itself. A 7-billion-parameter Qwen model learned tool use and science tasks this way. Researchers reported results comparable to full fine-tuning on both tasks. Tool-use accuracy was 69.07% with deletion tuning and 68.04% with full fine-tuning; science accuracy was 70.61% and 71.20%, respectively. In supervised fine-tuning and self-distillation, deletion masks were 600~4700x smaller than full fine-tuning checkpoints.
The model also solved new problems in evaluations with multiple generated answers. With 128 answers sampled per problem at temperature 1.0, the 8-billion-parameter Llama model's GSM8K solve rate was 99.62% with deletion tuning and 95.29% for the base model. This measures problems solved across multiple attempts, not single-answer accuracy.
4. Small storage comes at a training memory cost
In the main reinforcement learning comparison, deletion tuning masks were 3600~4700x smaller than full fine-tuning checkpoints. Switching tasks requires loading a small mask and applying it to the original weights. Despite lower storage needs, training assigns a trainable variable to each targeted weight, so training memory use is below full fine-tuning but above LoRA.
Retention of prior tasks is another advantage. Across three Llama model sizes, average accuracy on six tasks stayed within 1 percentage point of the base model throughout reinforcement learning. By contrast, LoRA and training that changes only a randomly selected subset of weights lost up to 5.4 percentage points.
Experiments on continual learning across successive tasks remain future work. These masks delete individual gate projection weights. Deleting structured groups to combine model adaptation with compression was also proposed as future work.
5. Learning what to delete mattered more than the initial choice
Selecting deletion positions once using Wanda scores produced results within 2.4 percentage points of the base model across six models. Learning masks from the same Wanda starting scores improved results by up to 53.2 percentage points over the base model. This suggests that training produced gains unavailable from the initial deletion choices alone.
Researchers also compared how broadly weight changes spread across directions, how much they altered the original matrix structure, and how much they affected important weights. Together, these three metrics placed deletion tuning closer to full fine-tuning than LoRA or random sparse fine-tuning, which adjusts only a selected subset of weights. The interpretation is that deleting few weights can still produce a pattern of change resembling full-weight training.
Deletion tuning's practical value lies in approaching large-model post-training accuracy while reducing both per-task storage needs and performance loss on prior tasks.