bioRxiv (Cold Spring Harbor Laboratory) · 2022 · 13 citations · 45 references
Life SciencesComputer ScienceNeuroscienceCognitive NeuroscienceComputational NeuroscienceMachine LearningBias-variance TradeoffClassificationFunctional NeuroimagingNeural Data AnalysisNeural DecodingNeural NetworksBinary ClassificationClass ImbalanceClass Imbalance LearningElectroencephalography DecodingImbalanceImbalanced ClassificationReceiver Operating Characteristic AnalysisVoting Classifiers
Abstract Machine learning (ML) is increasingly used in cognitive, computational and clinical neuroscience. The reliable and efficient application of ML requires a sound understanding of its subtleties and limitations. Training ML models on datasets with imbalanced classes is a particularly common problem, and it can have severe consequences if not adequately addressed. With the neuroscience ML user in mind, this paper provides a didactic assessment of the class imbalance problem and illustrates its impact through systematic manipulation of data imbalance ratios in (i) simulated data and (ii) brain data recorded with electroencephalography (EEG) and magnetoencephalography (MEG). Our results illustrate how the widely-used Accuracy (Acc) metric, which measures the overall proportion of successful predictions, yields misleadingly high performances, as class imbalance increases. Because Acc weights the per-class ratios of correct predictions proportionally to class size, it largely disregards the performance on the minority class. A binary classification model that learns to systematically vote for the majority class will yield an artificially high decoding accuracy that directly reflects the imbalance between the two classes, rather than any genuine generalizable ability to discriminate between them. We show that other evaluation metrics such as the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), and the less common Balanced Accuracy (BAcc) metric – defined as the arithmetic mean between sensitivity and specificity, provide more reliable performance evaluations for imbalanced data. Our findings also highlight the robustness of Random Forest (RF), and the benefits of using stratified cross-validation and hyperprameter optimization to tackle data imbalance. Critically, for neuroscience ML applications that seek to minimize overall classification error, we recommend the routine use of BAcc, which in the specific case of balanced data is equivalent to using standard Acc, and readily extends to multi-class settings. Importantly, we present a list of recommendations for dealing with imbalanced data, as well as open-source code to allow the neuroscience community to replicate and extend our observations and explore alternative approaches to coping with imbalanced data.
45
Leo Breiman · Machine Learning · 2001 · 126.2K citations · Full text
Corinna Cortes, Vladimir Vapnik · Machine Learning · 1995 · 40.5K citations · Full text
Matplotlib: A 2D Graphics Environment
John D. Hunter · Computing in Science & Engineering · 2007 · 39.6K citations
SMOTE: Synthetic Minority Over-sampling Technique
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence Hall et al. · Journal of Artificial Intelligence Research · 2002 · 31.2K citations · Full text
MEG and EEG data analysis with MNE-Python
Alexandre Gramfort · Frontiers in Neuroscience · 2013 · 4K citations · Full text