A Pathologist-Annotated Dataset for Validating Artificial Intelligence:\n A Project Description and Pilot Study

Abstract

Purpose: In this work, we present a collaboration to create a validation\ndataset of pathologist annotations for algorithms that process whole slide\nimages (WSIs). We focus on data collection and evaluation of algorithm\nperformance in the context of estimating the density of stromal tumor\ninfiltrating lymphocytes (sTILs) in breast cancer. Methods: We digitized 64\nglass slides of hematoxylin- and eosin-stained ductal carcinoma core biopsies\nprepared at a single clinical site. We created training materials and workflows\nto crowdsource pathologist image annotations on two modes: an optical\nmicroscope and two digital platforms. The workflows collect the ROI type, a\ndecision on whether the ROI is appropriate for estimating the density of sTILs,\nand if appropriate, the sTIL density value for that ROI. Results: The pilot\nstudy yielded an abundant number of cases with nominal sTIL infiltration.\nFurthermore, we found that the sTIL densities are correlated within a case, and\nthere is notable pathologist variability. Consequently, we outline plans to\nimprove our ROI and case sampling methods. We also outline statistical methods\nto account for ROI correlations within a case and pathologist variability when\nvalidating an algorithm. Conclusion: We have built workflows for efficient data\ncollection and tested them in a pilot study. As we prepare for pivotal studies,\nwe will consider what it will take for the dataset to be fit for a regulatory\npurpose: study size, patient population, and pathologist training and\nqualifications. To this end, we will elicit feedback from the FDA via the\nMedical Device Development Tool program and from the broader digital pathology\nand AI community. Ultimately, we intend to share the dataset, statistical\nmethods, and lessons learned.\n