P2P botnet detection using behavior clustering & statistical tests

Abstract

Botnets are widely believed to be the most serious danger to the Internet. Most recent research on botnet detection focuses on centralized botnets and primarily relies on two assumptions: prior knowledge of potential C&C channels and capability of monitoring them. However, when botnets switch to a P2P (peer-to-peer) structure and utilize multiple protocols for C&C, the above assumptions no longer hold. Consequently, the detection of P2P botnets is more difficult. In this paper, we relax the above two assumptions and focus on C&C channel detection for P2P botnets that use multiple protocols (randomly chosen) for C&C. We first consider a clustering based node behavior profiling approach to capture the node behavior clusters in a network; we then propose two detection schemes using formal statistical tests on popular behavior clusters in this network. In brief, we detect C&C behavior by measuring its impact on one or more normal behavior clusters in a statistical way. In the initial evaluations, we validate the assumptions made in this paper under different real user traces from enterprise network environments. We then evaluate the proposed approaches to detect the C&C channel in both simple and realistic cases and achieve encouraging results in terms of high detection and low false positive rates.