2021 FDA Science Forum
High Throughput Truthing (HTT) of Pathologist Annotations as a Reference Standard for Validating Artificial Intelligence in Digital Pathology
- Authors:
- Center:
-
Contributing OfficeCenter for Devices and Radiological Health
Abstract
Background
Recent advancements in whole slide imaging (WSI) technology have exponentially increased the development of tools in digital pathology analysis, including artificial intelligence (AI) algorithms, allowing pathologists to be more efficient and potentially improve their diagnoses. Before coming to market, AI algorithm performance must first be validated against a reference standard. There are few examples of creating a reference standard using pathologist-annotated ground truth. This project will fill this gap.
Purpose
This work harnesses the flexibility of the Medical Device Development Tool (MDDT) program and promotes innovative methods in tool design. The work will support good Machine Learning Practices for digital pathology and serve as a demonstration of quality data collection methods and accompanied statistical handling of algorithm performance with truth by clinician observation with no given truth.
Methodology
The clinical use case of this work is the evaluation of the density of tumor-infiltrating lymphocytes (TILs) in breast cancer tissue stained with hematoxylin and eosin. Pathologist annotations are collected through two modalities to compare pathologist agreement between the clinical standard technology, an optical microscope, and two digital platforms. The microscope was used to provide the reference standard, eliminating bias from slide digitization. Image data is sourced from multiple clinical sites, ensuring generalizable results. Multiple pathologists will annotate each image across platforms, allowing for intra- and inter- pathologist agreement analysis. Algorithm performance will compare algorithm-pathologist differences to pathologist-pathologist differences.
Results
Pilot data collected through the microscope system and through the digital platforms have led to critical evaluation of image and annotator parameters needed to design a generalizable dataset. The final dataset will include pathology glass slides, corresponding digital slide images, and location-specific annotations.
Conclusions
The dataset created through this work will be submitted for qualification as an MDDT. This effort creates a resource typically only available to large companies, promoting advancement of AI as a medical device for innovators of all sizes. Partnerships with external collaborators are a critical component of this work to reach community consensus on a standard reference dataset in digital pathology aligned with leading standards in clinical practice.
