Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning

TLDR

Federated learning may be vulnerable to poisoning attacks, but their real impact on production systems remains unclear. The study seeks to systematically catalog poisoning attacks on FL, clarify misconceptions, and provide guidelines for realistic research. The authors analyze untargeted poisoning attacks in realistic production FL settings, characterize threat models and adversarial capabilities, and propose new data and model poisoning attacks tested on three benchmark datasets. The study finds that FL is highly robust against poisoning attacks even with simple defenses, and demonstrates that newly proposed attacks are largely ineffective under such defenses across three benchmark datasets.

Abstract

While recent works have indicated that federated learning (FL) may be vulnerable to poisoning attacks by compromised clients, their real impact on production FL systems is not fully understood. In this work, we aim to develop a comprehensive systemization for poisoning attacks on FL by enumerating all possible threat models, variations of poisoning, and adversary capabilities. We specifically put our focus on un-targeted poisoning attacks, as we argue that they are significantly relevant to production FL deployments. We present a critical analysis of untargeted poisoning attacks under practical, production FL environments by carefully characterizing the set of realistic threat models and adversarial capabilities. Our findings are rather surprising: contrary to the established belief, we show that FL is highly robust in practice even when using simple, low-cost defenses. We go even further and propose novel, state-of-the-art data and model poisoning attacks, and show via an extensive set of experiments across three benchmark datasets how (in)effective poisoning attacks are in the presence of simple defense mechanisms. We aim to correct previous misconceptions and offer concrete guidelines to conduct more accurate (and more realistic) research on this topic.