2024 · 47 citations · 20 references
ComputingBiomedical InformaticsHealth InformaticsBiomedical ImagingDiagnostic ImagingNatural Language ProcessingClinical Natural Language ProcessingLanguage ModelingMedical Image AnalysisMedical ImagingMultimodal AnalysisVisual Question AnsweringEmpirical EvaluationImage-text CorporaLarge Language ModelsMedical Visual Question AnsweringVisual Language
Recently, large language models (LLMs) have taken the spotlight in natural language processing. Further, integrating LLMs with vision enables the users to explore emergent abilities with multimodal data. Visual language models (VLMs), such as LLaVA, Flamingo, or CLIP, have demonstrated impressive performance on various visio-linguistic tasks. Consequently, there are enormous applications of large models that could be potentially used in the biomedical imaging field. Along that direction, there is a lack of related work to show the ability of large models to diagnose the diseases. In this work, we study the zero-shot and few-shot robustness of VLMs on the medical imaging analysis tasks. Our comprehensive experiments demonstrate the effectiveness of VLMs in analyzing biomedical images such as brain MRIs, microscopic images of blood cells, and chest X- rays. While VLMs can not outperform classic vision models like CNN or ResNet, it is worth noting that VLMs can serve as chat assistants to provide pre-diagnosis before making decisions without the need for retraining or finetuning stages.
20
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Backpropagation Applied to Handwritten Zip Code Recognition
Yann LeCun, Bernhard E. Boser, J. S. Denker et al. · Neural Computation · 1989 · 11.9K citations
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu et al. · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 7.1K citations