2015 · 207 citations · 138 references
ComputingComputer ScienceFeature EngineeringMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsMulti-task LearningSpeech RecognitionTransfer LearningCross-lingual TransferCross-lingual Transfer LearningDeep Neural Network Acoustic ModelsFeature TransferLanguage Processing ModelsModel AdaptationSpeech Pattern Recognition
Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
138
Sinno Jialin Pan, Qiang Yang · IEEE Transactions on Knowledge and Data Engineering · 2009 · 23.2K citations
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text