Evaluating the Performance of Machine Learning Algorithms on Construction Accidents: An Application of ROC Curves

Abstract

The significant amount of safety data being collected on construction sites provides a valuable source of information for researchers seeking to better understand the root causes of accidents using advanced-analysis methods such as data mining and machine learning techniques. However, evaluating the predictive performance of these methods can be challenging. While simple measures such as total accuracy have been widely used to assess the classification performance of these algorithms, studies show that relying solely on these individual measures can be misleading. One of the techniques that can be used to organize classifiers and visualize their performance is receiver operating characteristics (ROC) curves. ROC curves are commonly used in machine learning and data mining studies to improve medical decision making; however, their application in construction safety is minimal. This study evaluates the application of ROC curves to more accurately interpret the output of machine learning methods, because understanding the mechanism of ROC curves will improve the classification performance of machine learning methods (i.e., AdaBoost) when predicting the degree of construction injuries. To achieve this goal, more than 700 accident reports were collected from national databases and analyzed via multiple machine learning techniques. The findings of this study indicate that applying ROC curves can yield the selection of better classification criteria and can reduce the cost of classification errors. The method is particularly beneficial when the costs of errors are unequal or when the data is highly imbalanced.