Empirical analysis of practitioners' perceptions of test flakiness factors

Abstract

Summary Identifying the root causes of test flakiness is one of the challenges faced by practitioners during software testing. In other words, the testing of the software is hampered by test flakiness. Since the research about test flakiness in large‐scale software engineering is scarce, the need for an empirical case‐study where we can build a common and grounded understanding of the problem as well as relevant remedies that can later be evaluated in a large‐scale context is a necessity. This study reports the findings from a multiple‐case study. The authors conducted an online survey to investigate and catalogue the root causes of test flakiness and mitigation strategies. We attempted to understand how practitioners perceive test flakiness in closed‐source development, such as how they define test flakiness and what practitioners perceive can affect test flakiness. The perceptions of practitioners were compared with the available literature. We investigated whether practitioners' perceptions are reflected in the test artefacts such as what is the relationship between the perceived factors and properties of test artefacts. This study reported 19 factors that are perceived by professionals to affect test flakiness. These perceived factors are categorized as test code , system under test , CI/test infrastructure , and organization‐related . The authors concluded that some of the perceived factors in test flakiness in closed‐source development are directly related to non‐determinism, whereas other perceived factors concern different aspects, for example, lack of good properties of a test case, deviations from the established processes, and ad hoc decisions. Given a data set from investigated cases, the authors concluded that two of the perceived factors (i.e., test case size and test case simplicity) have a strong effect on test flakiness.