Alice in warningland: a large-scale field study of browser security warning effectiveness

TLDR

The study evaluates the effectiveness of browser security warnings and offers design recommendations based on empirical findings. The authors collected over 25 million in‑browser warning impressions via telemetry from Firefox and Chrome. Users ignored 10 % of Firefox and 25 % of Chrome malware/phishing warnings, 33 % of Firefox SSL warnings, yet 70.2 % of Chrome SSL warnings, showing that warnings can be effective and that user experience influences compliance.

Abstract

We empirically assess whether browser security warnings are as ineffective as suggested by popular opinion and previous literature. We used Mozilla Firefox and Google Chrome's in-browser telemetry to observe over 25 million warning impressions in situ. During our field study, users continued through a tenth of Mozilla Firefox's malware and phishing warnings, a quarter of Google Chrome's malware and phishing warnings, and a third of Mozilla Firefox's SSL warnings. This demonstrates that security warnings can be effective in practice; security experts and system architects should not dismiss the goal of communicating security information to end users. We also find that user behavior varies across warnings. In contrast to the other warnings, users continued through 70.2% of Google Chrome's SSL warnings. This indicates that the user experience of a warning can have a significant impact on user behavior. Based on our findings, we make recommendations for warning designers and researchers.