IEEE Transactions on Circuits and Systems for Video Technology · 2020 · 10 citations · 38 references
ComputingComputer ScienceEducational Data MiningMachine LearningNatural Language ProcessingDeep LearningNeural NetworksRepresentation LearningSemi-supervised LearningSupervised LearningMaster Model ConstructionModel CompositionModel Composition SemanticsModel EvolutionSemi-supervised ModelsSemi-supervised Representation LearningStudent Model Development
In this work, we propose to push the performance limit of semi-supervised learning on very small sets of labeled samples by developing a new method called learned model composition with critical sample look-ahead (LMCS). Training efficient deep neural networks on much smaller sets of labeled samples is a challenging problem. With a small labeled set, the initial network suffers from low accuracy. Based on this error-prone network, the subsequent semi-supervised learning process will be fragile and unstable. To address this issue, we propose to introduce a look-ahead master model to identify the correct direction of model evolution to effectively guide the semi-supervised learning process of the student model. Specifically, our proposed LMCS method explores two major ideas. First, it introduces a new learned model composition structure so that we can compose a more efficient master network from student models of past iterations through a network learning process. Second, we develop a new method, called confined maximum entropy search, to discover new critical samples near the model decision boundary and provide the master model with look-ahead access to these samples to enhance its guidance capability. Our extensive experimental results demonstrate that the proposed LMCS method outperforms the state-of-the-art semi-supervised learning methods, especially on small sets of labeled samples. For example, on the CIFAR-10 dataset, with a very small set of 80 labeled samples, our method outperforms Google's MixMatch method, reducing the error rate by more than 10%.
38
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky et al. · 2014 · 34.3K citations
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever et al. · arXiv (Cornell University) · 2013 · 5.7K citations · Full text
mixup: Beyond Empirical Risk Minimization
Hongyi Zhang, Moustapha Cissé, Yann Dauphin et al. · arXiv (Cornell University) · 2017 · 4.8K citations · Full text
Reading digits in natural images with unsupervised feature learning
Yuval Netzer · 2024 · 4.6K citations · Full text
Semi-Supervised Learning Literature Survey
Xiaojin Zhu · Minds at UW (University of Wisconsin) · 2005 · 3.9K citations · Full text