Sapienz: multi-objective automated testing for Android applications

TLDR

Sapienz is a multi‑objective search‑based Android testing approach that automatically explores and optimises test sequences to minimise length while maximizing coverage and fault detection. Sapienz combines random fuzzing, systematic and search‑based exploration, using seeding and multi‑level instrumentation. Sapienz outperforms Dynodroid and Android Monkey in coverage, fault detection, and fault‑revealing sequence length, discovered 558 unique crashes in the top 1,000 Google Play apps, and has led to developer confirmation of 14 real faults, six of which already have fixes.

Abstract

We introduce Sapienz, an approach to Android testing that uses multi-objective search-based testing to automatically explore and optimise test sequences, minimising length, while simultaneously maximising coverage and fault revelation. Sapienz combines random fuzzing, systematic and search-based exploration, exploiting seeding and multi-level instrumentation. Sapienz significantly outperforms (with large effect size) both the state-of-the-art technique Dynodroid and the widely-used tool, Android Monkey, in 7/10 experiments for coverage, 7/10 for fault detection and 10/10 for fault-revealing sequence length. When applied to the top 1,000 Google Play apps, Sapienz found 558 unique, previously unknown crashes. So far we have managed to make contact with the developers of 27 crashing apps. Of these, 14 have confirmed that the crashes are caused by real faults. Of those 14, six already have developer-confirmed fixes.