Protocol state fuzzing of TLS implementations

Abstract

We describe a largely automated and systematic analysis of TLS implementations by what we call ‘protocol state fuzzing’: we use state machine learning to infer state ma-chines from protocol implementations, using only black-box testing, and then inspect the inferred state machines to look for spurious behaviour which might be an indica-tion of flaws in the program logic. For detecting the pres-ence of spurious behaviour the approach is almost fully automatic: we automatically obtain state machines and any spurious behaviour is then trivial to see. Detecting whether the spurious behaviour introduces exploitable security weaknesses does require manual investigation. Still, we take the point of view that any spurious func-tionality in a security protocol implementation is danger-ous and should be removed. We analysed both server- and client-side implemen-tations with a test harness that supports several key ex-change algorithms and the option of client certificate au-thentication. We show that this approach can catch an interesting class of implementation flaws that is appar-ently common in security protocol implementations: in three of the TLS implementations analysed new security flaws were found (in GnuTLS, the Java Secure Socket Extension, and OpenSSL). This shows that protocol state fuzzing is a useful technique to systematically analyse security protocol implementations. As our analysis of different TLS implementations resulted in different and unique state machines for each one, the technique can also be used for fingerprinting TLS implementations. 1