2024 · 22 citations · 33 references
ComputingComputer ScienceRadiologyComputer VisionDiagnostic RadiologyNatural Language ProcessingClinical Natural Language ProcessingDeep LearningLanguage ModelingMedical ImagingRepresentation LearningSelf-supervised LearningAutomated Report GenerationDeep Learning In RadiologyEmbedding LearningLarge Language ModelsMultimodal Representation LearningMultimodal Self-supervised LearningRadiology ReportingReport Generation From Chest X-rays
Chest X-Ray (CXR) images play a crucial role in clinical practice, providing vital support for diagnosis and treatment. Augmenting the CXR dataset with synthetically generated CXR images annotated with radiology reports can enhance the performance of deep learning models for various tasks. However, existing studies have primarily focused on generating unimodal data of either images or reports. In this study, we propose an integrated model, CXR-IRGen, designed specifically for generating CXR image-report pairs. Our model follows a modularized structure consisting of a vision module and a language module. Notably, we present a novel prompt design for the vision module by combining both text embedding and image embedding of a reference image. Additionally, we propose a new CXR report generation model as the language module, which effectively leverages a large language model and self-supervised learning strategy. Experimental results demonstrate that our new prompt is capable of improving the general quality (FID) and clinical efficacy (AUROC) of the generated images, with average improvements of 15.84% and 1.84%, respectively. Moreover, the proposed CXR report generation model outperforms baseline models in terms of clinical efficacy (F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> score) and exhibits a high-level alignment of image and text, as the best F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> score of our model is 6.93% higher than the state-of-the-art CXR report generation model. Our code is available at https://github.com/junjie-shentu/CXR-IRGen.
33
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh et al. · IEEE Transactions on Image Processing · 2004 · 56K citations
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Auto-Encoding Variational Bayes
Diederik P. Kingma, Max Welling · UvA-DARE (University of Amsterdam) · 2013 · 15.6K citations · Full text