State-of-Practice in GUI-based System and Acceptance Testing: An Industrial Multiple-Case Study

TLDR

Software testing evaluates quality, and system and acceptance tests validate high‑level conformance by executing end‑user GUI scenarios, yet no empirical studies have examined how such GUI‑based testing is performed in industry. This multiple‑case study investigates the state of practice of GUI‑based system and acceptance testing across six software development companies. The study finds that manual GUI‑based system testing is common, automated testing is limited, and key challenges include tool limitations, high costs, and customer involvement.

Abstract

Software testing is an essential means of evaluating software quality. System and acceptance tests aim to validate a system's conformance to its requirements on a high level of system abstraction. Therefore, they are generally performed by executing end-user scenarios through the system's graphical user interface (GUI). However, to the authors' best knowledge, there are no empirical studies that evaluate how GUI-based system and acceptance testing is performed in industrial practice. In this paper, we present a multiple-case study with the goal to investigate the state-of-practice of GUI-based system and acceptance testing at six software development companies of varying context. The main findings are that manual, GUI-based system testing is widespread and that automated GUI-based system and acceptance testing exists only on a small scale. Additionally, the study identifies core problems with GUI-based system and acceptance testing such as test tool limitations, high test costs and customer involvement in testing.