Publication | Closed Access
What is being transferred in transfer learning
44
Citations
0
References
2020
Year
Few-shot LearningConvolutional Neural NetworkEngineeringMachine LearningLoss LandscapeBlock-shuffled ImagesEducationCognitionLanguage LearningData SciencePattern RecognitionLearning ProblemFeature LearningLearning SciencesComputer EngineeringLearning AnalyticsComputer ScienceDeep LearningComputer VisionLearning TheoryDomain AdaptationTransfer Learning
One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.