Automated Oracle Comparators for TestingWeb Applications

Abstract

Software developers need automated techniques to maintain the correctness of complex, evolving Web applications. While there has been success in automating some of the testing process for this domain, there exists little automated support for verifying that the executed test cases produce expected results. We assist in this tedious task by presenting a suite of automated oracle comparators for testing Web applications. To effectively identify failures, each comparator is specialized to particular characteristics of the possibly nondeterministic Web applications' output in the form of HTML responses. We also describe combinations of comparators designed to achieve both high precision and recall in failure detection and a tool for helping testers to analyze the output of multiple oracles in detail. We present results from an evaluation of the effectiveness and costs of the oracle comparators. We also provide recommendations to testers on applying effective oracle comparators based on their application's characteristics.