Detecting Subverted Cryptographic Protocols by Entropy Checking

Abstract

What happens when your implementation of SSL or some other cryptographic protocol is subverted through a buffer overflow attack? You have been hacked, yes. Unfortunately, you may be unaware of it: because normal traffic is encrypted, most IDSs cannot monitor it. We propose a simple, yet efficient technique to detect most of such attacks, by computing the entropy of the flow and comparing it against known thresholds.