Evaluating Machine Learning-Based Classification Approaches: A New Method for Comparing Classifiers Applied to Human Driver Prediction Intentions

Abstract

In this research, a new performance assessment based on the Probability of Detection (POD) reliability measure is developed integrating and discussing the effect of further parameters on classification results and therefore establishing a new connection between relevant process parameters and the related classifier evaluation. To illustrate the introduced approach, machine learning-based recognition of complex driving situations for human drivers is interpreted. Based on sensor signals and a complex driving scenario, related dynamical changes are classified and compared using the POD approach. Based on the POD-related evaluation, different machine learning approaches can be clearly distinguished with respect to their ability to predict the correct driver behavior as a function of time prior to the event itself. The introduced approach allows a very detailed comparison of classifiers related to the effects of parameters affecting the processes to be classified. In addition to recently published results on this novel approach, an extension of the POD approach by considering false positives and varying decision threshold in the comparison process is proposed. Generalization of the introduced approach for binary and continuous data is presented.