iTranslated by AI
Method for Creating High-Performance AI in Real-World Environments: Distillation [Paper Summary]
Introduction
Hello, I'm mita, an aspiring AI engineer!
Today, I'll summarize and discuss research on DisTaC: Distillation for Task-vector Conditioning, a distillation method to make model merging work reliably in real-world environments.
While model merging is convenient, it often fails unexpectedly. This study clarifies the issues and solves them with lightweight distillation.
I hope sharing what I've learned will be helpful to you!
Paper Summary
- Model merging is a technique that combines multiple fine-tuned models into one multi-task model without additional training.
- However, in practice, performance collapses due to differences in task-vector norms and low confidence of source models.
- The authors solve this with a lightweight distillation preprocessing method called DisTaC.
- DisTaC requires only unlabeled data and about 500 steps, improving all existing merging methods.
Paper Details
Background
- With the proliferation of large models and fine-tuned models, model merging—combining models to create new capabilities—has gained attention.
- While benchmarks show high performance, merging is surprisingly fragile in real-world settings.
Problems with Model Merging
The authors identify two failure modes:
-
Difference in task-vector norms
- Some tasks are overemphasized, suppressing the performance of others.
- → Occurs easily when learning rates or steps differ.
-
Low confidence of source models
- Ambiguous predictive distributions cause a significant drop in merged performance.
- → Worsened by label smoothing or Mixup.
Proposed Method: DisTaC
- Distillation for Task-vector Conditioning (DisTaC)
- Adjusts task vectors via distillation before merging:
- Aligns norms: Shrinks large vectors while preserving performance through distillation.
- Increases confidence: Distills the student model to be more overconfident than the teacher.
- Key features:
- Can be performed using only unlabeled data.
- Lightweight (about 500 steps).
- Compatible with any existing merging method.
Results
- Validated on ViT-B/L backbones with 8 visual tasks.
- Up to +20% accuracy improvement.
- Particularly for TSVM, achieves performance comparable to ideal merging conditions even under failure modes.
Practical Guidelines
- For norm adjustment, it is more stable to shorten long vectors than to lengthen short ones.
- Before merging, adjust models toward overconfidence; after merging, perform calibration.
What is Distillation?
- Knowledge Distillation: A method where a student model learns the teacher model's predictive distribution (soft targets).
- It transfers implicit knowledge, such as class relationships, better than simple label supervision.
- This study is novel in using distillation as preprocessing before merging.
Personal Impressions
Interpreting the Paper
Model merging, a well-known method for integrating AI models with different characteristics into a high-performance multimodal model, requires that the architectures of the models being merged are the same.
Additionally, it has drawbacks: when the difference in task vectors is large, or when the base model has low confidence, the performance after merging degrades significantly.
These issues occur frequently in real-world environments, raising questions about whether a model that performs well in papers can withstand practical use.
This study solves these problems using distillation as follows:
- Training to reduce the difference in task vectors without changing the input and output.
- Training the student model to be more overconfident than the teacher model.
By doing so, it succeeds in suppressing performance degradation while resolving the bottleneck issues.
Normally, distillation requires tens of thousands to hundreds of thousands of training steps, making it computationally expensive. However, by using pre-trained parameters as the initial values of the task vectors, this study achieves high performance with only about 500 steps, a lightweight load.
Discussion (Reflections)
Previously, a paper called VisCodex constructed a multimodal model capable of generating code from images by merging models with the same architecture.
When reading that work, I wondered, "In reality, isn't it difficult to align models with the same architecture?" This led me to read the current DisTaC paper.
What I originally hoped for was a method to unify different architectures, but this paper also deals with merging within the same architecture.
However, I feel this study offers a more practical model integration approach because it addresses problems that are likely to occur in real-world scenarios even with the same architecture (differences in task vectors or confidence calibration shifts).
Moreover, given that most other model merging papers also primarily experiment with the same architecture, my initial hypothesis that "in the real world, models of different architectures are more common" may itself be incorrect.
On the other hand, as an advanced application of this paper, one could consider using distillation to convert models of different architectures into the same architecture and then merge them. However, this is purely an amateur idea, and its feasibility is unknown.
First, I would like to investigate what kinds of architectures actually exist in the real world.
Discussion