Machine learning based natural language processing of radiology reports in orthopaedic trauma

TLDR

Assessing NLP performance is essential for clinical tasks such as avoiding missed injuries and quality checks, and for research tasks like cohort identification and radiograph annotation. The study compares different machine‑learning NLP methods for classifying orthopaedic trauma radiology reports to detect injuries. Dutch radiology reports of injured extremities (n = 2469, 33 % fractures) and chest radiographs (n = 799, 20 % pneumothorax) were labeled by radiologists and trauma surgeons, and NLP classification was performed with rule‑based, ML, and BERT models, optimizing preprocessing and evaluating performance with F1‑score, AUC, sensitivity, specificity, and accuracy. The BERT model achieved an F1‑score of 95 ± 2 % and accuracy of 96 ± 1 % on simple reports, and 83 ± 7 % F1 with 93 ± 2 % accuracy on complex reports, outperforming traditional ML and rule‑based classifiers.

Abstract

To compare different Machine Learning (ML) Natural Language Processing (NLP) methods to classify radiology reports in orthopaedic trauma for the presence of injuries. Assessing NLP performance is a prerequisite for downstream tasks and therefore of importance from a clinical perspective (avoiding missed injuries, quality check, insight in diagnostic yield) as well as from a research perspective (identification of patient cohorts, annotation of radiographs). Datasets of Dutch radiology reports of injured extremities (n = 2469, 33% fractures) and chest radiographs (n = 799, 20% pneumothorax) were collected in two different hospitals and labeled by radiologists and trauma surgeons for the presence or absence of injuries. NLP classification was applied and optimized by testing different preprocessing steps and different classifiers (Rule-based, ML, and Bidirectional Encoder Representations from Transformers (BERT)). Performance was assessed by F1-score, AUC, sensitivity, specificity and accuracy. The deep learning based BERT model outperforms all other classification methods which were assessed. The model achieved an F1-score of (95 ± 2)% and accuracy of (96 ± 1)% on a dataset of simple reports (n= 2469), and an F1 of (83 ± 7)% with accuracy (93 ± 2)% on a dataset of complex reports (n= 799). BERT NLP outperforms traditional ML and rule-base classifiers when applied to Dutch radiology reports in orthopaedic trauma.