Prevalence and Maintenance of Automated Functional Tests for Web Applications

Abstract

Functional testing requires executing particular sequences of user actions. Test automation tools enable scripting user actions such that they can be repeated more easily. SELENIUM, for instance, enables testing Web applications through scripts that interact with a Web browser and assert properties about its observable state. However, little is known about how common such tests are in practice. We therefore present a cross-sectional quantitative study of the prevalence of SELENIUM-based tests among open-source Web applications, and of the extent to which such tests are used within individual applications. Automating functional tests also brings about the problem of maintaining test scripts. As the system under test evolves, its test scripts are bound to break. Even less is known about the way test scripts change over time. We therefore also present a longitudinal quantitative study of whether and for how long test scripts are maintained, as well as a longitudinal qualitative study of the kind of changes they undergo. To the former's end, we propose two new metrics based on whether a commit to the application's version repository touches a test file. To the latter's end, we propose to categorize the changes within each commit based on the elements of the test upon which they operate. As such, we are able to identify the elements of a test that are most prone to change.