FIG: A Prototype Tool for Online Verification of Recovery Mechanisms

Abstract

Network applications of the future will require advanced mechanisms for automatic failure recovery in order to provide an acceptable quality of service. Because of this requirement, there is a need for tools that can inject simulated faults to verify a system’s fault detection and recovery methods. We present a set of objectives that such a fault injection tool must meet, and describe a prototype we have developed that meets these criteria. This tool, FIG, is capable of selectively introducing and logging errors at the application/library boundary on a running system. Our expectation is that FIG will be used for software development and verification, as well as for on-site testing of production systems. We use a working version of this tool to test the behavior of several common UNIX applications under simulated failures and offer suggestions from these tests on how to develop software with a higher degree of failure recoverability.