Monitoring machine learning models: a categorization of challenges and methods

Abstract

The importance of software based on machine learning is growing rapidly, but the potential of prototypes may not be realized in operation. This study identified six categories of challenges for verification and validation of machine learning applications during production. Subsequently, monitoring was analyzed as a possible solution to mitigate those challenges. Capturing relevant data and model metrics may reveal problems at an early stage, allowing for targeted countermeasures. This study presents a taxonomy of methods and metrics currently addressed in scientific literature and compares these categories with case studies from practice.