2020 · 43 citations · 33 references
ComputingAutonomous SystemsComputer ScienceArtificial IntelligenceAutonomous VehiclesMachine LearningConfidence CalibrationNeural NetworksPropagation Of UncertaintyUncertainty AnalysisUncertainty EstimationUncertainty RepresentationComplexityComplexity Trade-offsMetricsModel ComplexityModel UncertaintyModel Uncertainty EstimationSelective PredictionUncertainty Characterization
Accurate uncertainty estimation in neural network predictions is essential for trusted DNN models, with growing interest across tasks such as security cameras and autonomous driving. This study focuses on selective prediction and confidence calibration as the two primary use cases of uncertainty estimation. We identify shortcomings of existing quality metrics for these use cases, introduce new metrics to address them, and apply them to examine how model complexity affects uncertainty estimation quality. Experiments confirm the proposed metrics outperform existing ones, revealing notable trends in the complexity–uncertainty trade‑off.
Accurately estimating uncertainties in neural network predictions is of great importance in building trusted DNNs-based models, and there is an increasing interest in providing accurate uncertainty estimation on many tasks, such as security cameras and autonomous driving vehicles. In this paper, we focus on the two main use cases of uncertainty estimation, i.e., selective prediction and confidence calibration. We first reveal potential issues of commonly used quality metrics for uncertainty estimation in both use cases, and propose our new metrics to mitigate them. We then apply these new metrics to explore the trade-off between model complexity and uncertainty estimation quality, a critically missing work in the literature. Our empirical experiment results validate the superiority of the proposed metrics, and some interesting trends about the complexity-uncertainty trade-off are observed.
33
Densely Connected Convolutional Networks
Gao Huang, Zhuang Liu, Laurens van der Maaten et al. · 2017 · 44.7K citations
The relationship between Precision-Recall and ROC curves
Jesse Davis, Mark Goadrich · 2006 · 6.3K citations · Full text
Sergey Zagoruyko, Nikos Komodakis · 2016 · 6K citations