SpotTune: Transfer Learning Through Adaptive Fine-Tuning

TLDR

Transfer learning, which lets a source task influence the inductive bias of a target task, is widely used in computer vision, typically by fine‑tuning a pretrained deep neural network on target data. This paper proposes SpotTune, an adaptive fine‑tuning approach that selects the optimal fine‑tuning strategy per instance for target data. SpotTune employs a policy network that routes each target image through either fine‑tuned or pretrained layers, and extensive experiments demonstrate its effectiveness. SpotTune outperforms traditional fine‑tuning on 12 of 14 standard datasets, surpasses other state‑of‑the‑art strategies, and achieves the highest score on the Visual Decathlon dataset.

Abstract

Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pretrained on the source task using data from the target task. In this paper, we propose an adaptive fine-tuning approach, called SpotTune, which finds the optimal fine-tuning strategy per instance for the target data. In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers. We conduct extensive experiments to demonstrate the effectiveness of the proposed approach. Our method outperforms the traditional fine-tuning approach on 12 out of 14 standard datasets. We also compare SpotTune with other state-of-the-art fine-tuning strategies, showing superior performance. On the Visual Decathlon datasets, our method achieves the highest score across the board without bells and whistles.