To buy or not to buy—evaluating commercial AI solutions in radiology (the ECLAIR guidelines)

TLDR

Artificial intelligence has advanced rapidly in medical imaging, leading to many commercial AI products that radiology practices must evaluate, yet few guidelines exist to help assess these complex tools. The authors aim to develop the ECLAIR guidelines, a practical framework to help stakeholders evaluate and decide on purchasing commercial AI solutions in radiology. The framework outlines key evaluation topics—clinical relevance, performance and validation, usability and integration, regulatory and legal compliance, and financial and support services—to guide stakeholders. Using the ECLAIR framework, stakeholders can systematically evaluate AI solutions across relevance, performance, usability, regulatory, and financial dimensions, leading to informed purchase decisions.

Abstract

Abstract Artificial intelligence (AI) has made impressive progress over the past few years, including many applications in medical imaging. Numerous commercial solutions based on AI techniques are now available for sale, forcing radiology practices to learn how to properly assess these tools. While several guidelines describing good practices for conducting and reporting AI-based research in medicine and radiology have been published, fewer efforts have focused on recommendations addressing the key questions to consider when critically assessing AI solutions before purchase. Commercial AI solutions are typically complicated software products, for the evaluation of which many factors are to be considered. In this work, authors from academia and industry have joined efforts to propose a practical framework that will help stakeholders evaluate commercial AI solutions in radiology (the ECLAIR guidelines) and reach an informed decision. Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services. Key Points • Numerous commercial solutions based on artificial intelligence techniques are now available for sale, and radiology practices have to learn how to properly assess these tools. • We propose a framework focusing on practical points to consider when assessing an AI solution in medical imaging, allowing all stakeholders to conduct relevant discussions with manufacturers and reach an informed decision as to whether to purchase an AI commercial solution for imaging applications. • Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services.