Why We Should Have Seen That Coming

TLDR

Microsoft shut down the Tay chatbot within 24 hours after it posted racist, sexist, and anti‑Semitic tweets. This paper examines the Tay chatbot launched in March 2016 and argues that its rapid failure exposes fundamental issues with learning software that interacts with the public and the heightened responsibilities of its developers. The Tay incident demonstrates that learning software interacting with people or social media imposes additional ethical duties on developers, creating a greater burden of care beyond standard software practices.

Abstract

In this paper we examine the case of Tay, the Microsoft AI chatbot that was launched in March, 2016. After less than 24 hours, Microsoft shut down the experiment because the chatbot was generating tweets that were judged to be inappropriate since they included racist, sexist, and anti-Semitic language. We contend that the case of Tay illustrates a problem with the very nature of learning software (LS is a term that describes any software that changes its program in response to its interactions) that interacts directly with the public, and the developer's role and responsibility associated with it. We make the case that when LS interacts directly with people or indirectly via social media, the developer has additional ethical responsibilities beyond those of standard software. There is an additional burden of care.