Nature Communications · 2025 · 129 citations · 54 references
ComputingBiomedical InformaticsComputer ScienceBiomedical Text MiningMachine LearningNatural Language ProcessingLanguage ModelingLanguage ModelsNatural Language UnderstandingQuestion AnsweringClinical Question AnsweringEvaluation BenchmarksFew-shot LearningFine-tuningLarge Language ModelsLarge-scale Natural Language ProcessingManual Annotation CurationManual Curation
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs-GPT and LLaMA representatives-on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
54
SciPy 1.0: fundamental algorithms for scientific computing in Python
Nature Methods · 2020 · 37.9K citations · Full text
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang et al. · arXiv (Cornell University) · 2022 · 4.3K citations · Full text
Large language models encode clinical knowledge
Karan Singhal, Shekoofeh Azizi, Tao Tu et al. · Nature · 2023 · 3.2K citations · Full text
Computing, Health Informatics, Computational Linguistics +16