Multimodal Knowledge Distillation for VQA
Distilled a ViLT VQA model into a ~half-size student (6 layers, hidden size 384) under four losses — logit, hidden-state, attention, and hybrid — trained and evaluated through one shared harness on VQA v1.
- PyTorch
- ViLT
- Knowledge Distillation
- VQA
- Hugging Face
Shrinking ViLT four different ways
The teacher is ViLT (vilt-b32-finetuned-vqa), a single-stream vision-language transformer, fine-tuned for VQA as a 3,129-way multi-label classifier (an image can have several correct answers, weighted by how many annotators gave each one). The student is the same architecture at roughly half size: 6 transformer layers and a hidden size of 384, versus the teacher's 12 layers and 768. It trains from scratch under four different distillation losses on VQA v1 (MSCOCO train2014/val2014 images, 10% subsample for compute).
The four losses
- Logit distillation. KL-divergence between student and teacher output distributions, softened at temperature 3.0.
- Hidden-state distillation. MSE between the student's first hidden layer and the teacher's, after a learned linear projector maps the teacher's 768-dim states down to the student's 384.
- Attention distillation. MSE between student and teacher attention maps, with the teacher's 12 layers reduced to the student's 6 either by taking the first 6 directly or by averaging teacher layers in pairs.
- Hybrid distillation. All three losses summed into one, each still added to the student's own task loss at a tunable interpolation weight.
Each strategy trains its own student checkpoint under a shared harness (same teacher, data, optimizer, and top-1/top-5 accuracy tracking), so the four are directly comparable once training runs to completion. That comparison is the open next step. What's here is the four working loss implementations and the shared evaluation harness they run under.