The Oracle Problem in Software Testing: A Survey

TLDR

Testing seeks to uncover faults by comparing observed system behavior to expected outcomes, a challenge known as the test oracle problem, for which literature proposes techniques such as modelling, specifications, contract‑driven development, and metamorphic testing, yet when these are insufficient humans provide informal guidance and all oracle approaches must balance cost and benefit. This paper surveys current test oracle automation approaches and analyzes trends to address the bottleneck that limits broader test automation. The study reviews existing methods for automating test oracles, highlighting how automation replaces manual human judgment of correct behavior.

Abstract

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the "test oracle problem". Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.