On Chances and Risks of Security Related Algorithmic Decision Making Systems

Abstract

Recently, various decisions in security-related processes are assisted by so-called algorithmic decision making (ADM) systems, e.g., for predicting recidivism rates of criminals, for assessing the risk of a person being a terrorist, or the prediction of future criminal acts (predictive policing). However, the quality of such risk assessment is dependent on many modeling decisions. Based on requirements of proper democratic processes, especially security related ADM systems might thus require societal oversight. We argue that based on democracy-based processes it also needs to be discussed and decided, how aspects of its quality should be assessed: e.g., neither the proper measure for racial bias nor the one for its overall accuracy of prediction is decided on today. Finally, even if the ADM system would be as objective and perfect as it can be, its embedding in an important societal process might have severe side effects and needs to be controlled. In this article, we analyze the situation based on a political science view. We then point to some crucial decisions that need to be made in the planning stage, questions that need to be asked when purchasing a system, and measures that need to be implemented to measure the overall quality of the societal process in which the system is embedded in.