Token-Pruned Dynamic ViT with LoRA Adaptation for Efficient On-Device Inference
Compression and fine-tuning of Vision Transformer (ViT) architectures for edge devices. The pipeline reduces the quadratic complexity of self-attention \(\mathcal{O}(N^2)\) by pruning non-informative tokens at intermediate layers 4, 7, and 10, while adapting weights via LoRA.
Max Pruning at L10
91%
Only 9% of tokens retained at the final layer (keep_rate = 0.09).
Accuracy after LoRA Finetuning
~65%
Accuracy achieved quickly on the STL-10 test set.
FLOPs Reduction
-33%
The model goes from 0.18 GFLOPs (Teacher) to 0.12 GFLOPs (Pruned Student).
Low Parameter Size
1.56M
Reduced volume of parameters to fit edge constraints.
Image Patch Retention Profile by Layer
Self-Attention Mechanism & Adaptive Pruning
The base equation for MSA attention blocks is calculated over all patches of the cropped image:
To stabilize convergence and avoid early performance collapse, the pipeline deploys an adaptive sigmoidal pruning schedule that orchestrates the retention rate smoothly across epochs.
Project Report
Browse the validation and reproducibility document for Dynamic ViT with LoRA adaptation.