Information Fusion · 2023 · 38 citations · 60 references
ComputingComputer ScienceHealth InformaticsBiomedical Signal ProcessingMachine LearningUncertainty QuantificationClassificationDeep LearningElectrocardiogram AnalysisElectrocardiographyEnsemble LearningEnsemble MethodsAutomated Ecg InterpretationClassification AccuracyComputer-aided Electrocardiogram InterpretationDeep EnsemblesExternal Validation CohortsExternal Validation Of SegmentsMulti-label Ensemble LearningRobust Uncertainty Analysis
Artificial Intelligence (AI) use in automated Electrocardiogram (ECG) classification has continuously attracted the research community’s interest, motivated by their promising results. Despite their great promise, limited attention has been paid to the robustness of their results, which is a key element for their implementation in clinical practice. Uncertainty Quantification (UQ) is a critical for trustworthy and reliable AI, particularly in safety-critical domains such as medicine. Estimating uncertainty in Machine Learning (ML) model predictions has been extensively used for Out-of-Distribution (OOD) detection under single-label tasks. However, the use of UQ methods in multi-label classification remains underexplored. This study goes beyond developing highly accurate models comparing five uncertainty quantification methods using the same Deep Neural Network (DNN) architecture across various validation scenarios, including internal and external validation as well as OOD detection, taking multi-label ECG classification as the example domain. We show the importance of external validation and its impact on classification performance, uncertainty estimates quality, and calibration. Ensemble-based methods yield more robust uncertainty estimations than single network or stochastic methods. Although current methods still have limitations in accurately quantifying uncertainty, particularly in the case of dataset shift, incorporating uncertainty estimates with a classification with a rejection option improves the ability to detect such changes. Moreover, we show that using uncertainty estimates as a criterion for sample selection in active learning setting results in greater improvements in classification performance compared to random sampling.
60
Advances in Neural Information Processing Systems 29
Onur Teymur, Kostas Zygalakis, Ben Calderhead · 2016 · 13.5K citations
Markus Breunig, Hans‐Peter Kriegel, Raymond T. Ng et al. · ACM SIGMOD Record · 2000 · 5.2K citations
New Effect Size Rules of Thumb
Shlomo S. Sawilowsky · Journal of Modern Applied Statistical Methods · 2009 · 3.2K citations · Full text