arXiv (Cornell University) · 2020 · 278 citations · 30 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningMedical Image AnalysisMedical ImagingRepresentation LearningUnsupervised LearningContrastive LearningFine-tuningMedical Image ClassificationPretrainingTransferable Representation LearningUnsupervised Transfer LearningZero-shot Image RetrievalZero-shot Retrieval
Progress in learning visual representations for medical images is limited by scarce human annotations, and existing approaches that fine‑tune ImageNet weights or extract labels from reports are suboptimal. The authors introduce ConVIRT, an unsupervised method that learns medical image representations by leveraging naturally paired descriptive text. ConVIRT pretrains image encoders using a bidirectional contrastive objective between images and their paired text, a domain‑agnostic approach that requires no expert input. When transferred to four classification tasks and two zero‑shot retrieval tasks, ConVIRT outperforms strong baselines and attains comparable or better performance with only 10 % of the labeled data, demonstrating superior data efficiency.
Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we find these methods help little on medical images because of their high inter-class similarity. We propose ConVIRT, an alternative unsupervised strategy to learn medical visual representations by exploiting naturally occurring paired descriptive text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test ConVIRT by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that it leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10\% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.
30
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Momentum Contrast for Unsupervised Visual Representation Learning
Kaiming He, Haoqi Fan, Yuxin Wu et al. · 2020 · 12K citations
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Kelvin Xu, Jimmy Ba, Ryan Kiros et al. · arXiv (Cornell University) · 2015 · 7.5K citations · Full text