Automated Experiments on Ad Privacy Settings

TLDR

Google’s Ad Settings webpage was created to give users information about and some choice over the profiles Google builds, though the authors cannot determine who caused the observed effects due to limited visibility into the ad ecosystem. The study introduces AdFisher, an automated tool designed to explore interactions among user behavior, Google ads, and Ad Settings. AdFisher conducts browser-based experiments and applies machine learning and significance tests within a rigorous experimental design to ensure statistical soundness. AdFisher revealed that Ad Settings is opaque about certain profile features, offers limited choice that can produce seemingly discriminatory ads—e.g., substance‑abuse site visits altered ads but not settings, and setting gender to female reduced high‑pay job ads—providing a basis for further investigation.

Abstract

Abstract To partly address people’s concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google’s ads, and Ad Settings interact. AdFisher can run browser-based experiments and analyze data using machine learning and significance tests. Our tool uses a rigorous experimental design and statistical analysis to ensure the statistical soundness of our results. We use AdFisher to find that the Ad Settings was opaque about some features of a user’s profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse changed the ads shown but not the settings page. We also found that setting the gender to female resulted in getting fewer instances of an ad related to high paying jobs than setting it to male. We cannot determine who caused these findings due to our limited visibility into the ad ecosystem, which includes Google, advertisers, websites, and users. Nevertheless, these results can form the starting point for deeper investigations by either the companies themselves or by regulatory bodies.