Predicting treatment response from longitudinal images using multi-task deep learning

TLDR

Radiographic imaging is routinely used to evaluate treatment response in solid tumors, yet current imaging response metrics do not reliably predict the underlying biological response. The study introduces a multi‑task deep‑learning approach that simultaneously segments tumors and predicts treatment response. The method employs two Siamese subnetworks joined at multiple layers to integrate multi‑scale feature representations and compare pre‑ and post‑treatment images, trained on 2,568 MRI scans from 321 rectal cancer patients to predict pathologic complete response after neoadjuvant chemoradiotherapy. In multi‑institution validation, the imaging‑based model achieved AUCs of 0.95 and 0.92 in two independent cohorts, and when combined with blood‑based tumor markers the integrated model reached an AUC of 0.97, indicating that the approach may be broadly applied for screening, treatment response evaluation, disease monitoring, and surveillance.

Abstract

Abstract Radiographic imaging is routinely used to evaluate treatment response in solid tumors. Current imaging response metrics do not reliably predict the underlying biological response. Here, we present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction. We design two Siamese subnetworks that are joined at multiple layers, which enables integration of multi-scale feature representations and in-depth comparison of pre-treatment and post-treatment images. The network is trained using 2568 magnetic resonance imaging scans of 321 rectal cancer patients for predicting pathologic complete response after neoadjuvant chemoradiotherapy. In multi-institution validation, the imaging-based model achieves AUC of 0.95 (95% confidence interval: 0.91–0.98) and 0.92 (0.87–0.96) in two independent cohorts of 160 and 141 patients, respectively. When combined with blood-based tumor markers, the integrated model further improves prediction accuracy with AUC 0.97 (0.93–0.99). Our approach to capturing dynamic information in longitudinal images may be broadly used for screening, treatment response evaluation, disease monitoring, and surveillance.