Deep learning based tissue analysis predicts outcome in colorectal cancer

TLDR

Image-based machine learning, especially deep learning, has recently achieved expert-level accuracy in medical image classification. The study trains a deep network that combines convolutional and recurrent architectures to predict colorectal cancer outcomes from tumour tissue images. The model directly predicts patient outcome from small H&E‑stained tissue microarray images using a convolutional‑recurrent deep network, evaluated on 420 colorectal cancer patients with available clinicopathological data. The model achieved a hazard ratio of 2.3 (95 % CI 1.79–3.03) and AUC 0.69, outperforming expert visual assessment (HR 1.67–1.65, AUC 0.58–0.57) and demonstrating that deep learning extracts more prognostic information from colorectal cancer tissue morphology than experienced observers.

Abstract

Abstract Image-based machine learning and deep learning in particular has recently shown expert-level accuracy in medical image classification. In this study, we combine convolutional and recurrent architectures to train a deep network to predict colorectal cancer outcome based on images of tumour tissue samples. The novelty of our approach is that we directly predict patient outcome, without any intermediate tissue classification. We evaluate a set of digitized haematoxylin-eosin-stained tumour tissue microarray (TMA) samples from 420 colorectal cancer patients with clinicopathological and outcome data available. The results show that deep learning-based outcome prediction with only small tissue areas as input outperforms (hazard ratio 2.3; CI 95% 1.79–3.03; AUC 0.69) visual histological assessment performed by human experts on both TMA spot (HR 1.67; CI 95% 1.28–2.19; AUC 0.58) and whole-slide level (HR 1.65; CI 95% 1.30–2.15; AUC 0.57) in the stratification into low- and high-risk patients. Our results suggest that state-of-the-art deep learning techniques can extract more prognostic information from the tissue morphology of colorectal cancer than an experienced human observer.