Clinical performance pilot using cognitive computing for clinical trial matching at Mayo Clinic.

Abstract

e18598 Background: IBM Watson for Clinical Trial Matching (CTM) is a cognitive computing solution that uses natural language processing to match patients to clinical trials through analysis of structured and unstructured data from a patient’s Electronic Medical Record (EMR) against protocol eligibility criteria. This pilot was conducted collaboratively by Mayo Clinic, IBM Watson Health and Novartis to develop enhanced CTM functionality, allowing a global evaluation of patients at a site to determine the feasibility of accrual to individual clinical trials. Methods: We identified 7 ongoing or completed Novartis protocols1 (breast n = 4; lung n = 3) and obtained EMR patient data from a date-specific sampling of 376 breast and lung cancer patients seen at the Mayo Clinic in 2017. Using trial eligibility criteria and filters for patient setting (metastatic vs. non-metastatic), tumor stage, genetic markers and histology, Mayo nurse abstractors manually reviewed charts and assigned patients as potentially eligible or ineligible for each protocol. Time to complete this review per patient was recorded. Overall predictive accuracy, sensitivity, specificity, and related metrics were derived. Results: Overall eligibility accuracy of the breast protocols was 87.6% Patient attribute accuracy included: metastatic, 90.4%; neoadjuvant, 87.2%; HER2 status, 93%; and hormone receptor status, 88.6% For lung protocols, overall eligibility accuracy was 74.9%. Patient attribute accuracy included: metastatic, 87.7%; stage IIIB, 74.1%; ALK mutation, 72.6%; EGFR mutation, 75.9%; and subtype (NSCLC or SCLC), 91.4%. Manual screening time for 376 patient cases totaled 441 minutes (7.4 hours). Conclusions: Using Watson CTM to evaluate clinical trial criteria against a pool of institutional patients is a rapid and effective way to assess site feasibility and identify cohorts of potentially eligible patients. Further development and deployment of volume-based screening is highly desirable given the increasing complexity of clinical trials. The overall higher eligibility accuracy observed for breast trials compared to lung may be related to an increased number of breast training iterations prior to the pilot study.