Bias in computer systems

TLDR

Bias in computer systems stems from preexisting social institutions, technical constraints, and emergent use contexts, yet no comprehensive framework has been offered to understand or remedy it. The study identifies three bias categories—preexisting, technical, and emergent—and argues that freedom from bias should be evaluated alongside reliability, accuracy, and efficiency when judging system quality.

Abstract

From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.