IEEE Transactions on Medical Imaging · 2024 · 16 citations · 17 references
ComputingBiomedical InformaticsHealth InformaticsClinical Decision SupportDiagnostic ReasoningNatural Language ProcessingClinical Natural Language ProcessingDecision Support In HealthcareDiagnostic Decision SupportInformation ExtractionQuestion AnsweringBiomedical Question AnsweringClinical Question AnsweringClinical Question Answering SystemsKey Information ExtractionLarge Language ModelsMultimodal Question AnsweringQuestion Answering Systems
Medical question answering aims to enhance diagnostic support, improve patient education, and assist in clinical decision-making by automatically answering medical-related queries, which is an important foundation for realizing intelligent healthcare. Existing methods predominantly focus on extracting key information from a single data source, e.g., CT image, for answering. However, these methods are not enough to promote the development of intelligent healthcare, because they lack comprehensive medical diagnosis capabilities, which usually require the integration of multi-source data (e.g., laboratory tests, radiology images, pathology images, etc.) for processing. To address these limitations, our paper introduces the extended task of medical question answering, named medical diagnosis question answering MedDQA. MedDQA task aims to answer questions related to medical diagnosis based on multi-source data. Specifically, we introduce a corresponding dataset that incorporates multi-source diagnostic information from 250,917 patients in clinical data from hospital records, and utilize a large-scale model for constructing Q&A pairs. We propose a novel system based on large language models, named medical multi-agent (MMA) system, which includes a mechanism of multiple agents to handle different medical tasks. Each agent is specifically tailored to process various modalities of data and provide outputs in a uniform textual modality. Experimental results demonstrate that the MMA system's architecture significantly enhances the handling of multi-source data, thereby improving medical diagnosis, establishing a robust baseline for future research.
17
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Hao Tan, Mohit Bansal · 2019 · 2.3K citations · Full text
Yogesh Kumar, Apeksha Koul, Ruchi Singla et al. · Journal of Ambient Intelligence and Humanized Computing · 2022 · 999 citations · Full text
A dataset of clinically generated visual questions and answers about radiology images
Jason J. Lau, Soumya Gayen, Asma Ben Abacha et al. · Scientific Data · 2018 · 402 citations · Full text
Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering
Bo Liu, Li-Ming Zhan, Li Xu et al. · 2021 · 233 citations