Of course I wouldn't do that in real life

Abstract

We offer a nuanced examination of the way that realism can impact internal and external validity in HCI experiments. We show that if an HCI experiment lacks realism across any of four dimensions--appearance, content, task and setting--the lack of realism can confound the study by interacting with the treatment and weakening internal or external validity. We argue furthermore, that realism can be increased while still maintaining control: analogue experiments allow researchers to conduct experiments in more ecologically valid environments and online experiments bridge the gap between the cleanroom and field. While increasing the level of realism in an experiment can introduce noise, technological developments have made it easier to collect rich analytics on behavior and usage.