Development of a benchmark to measure system robustness

Abstract

An initial attempt at the development of a set of benchmarks to gauge a system's robustness as measured by its ability to tolerate errors is presented. Due to the large domain of system components whose intolerance to errors can lead to system failure, several primitive benchmarks that can be combined into a robustness benchmark suite are presented. Each primitive benchmark targets a system functionality and measure its behavior given erroneous inputs. Four primitive benchmarks have been implemented in this initial effort. They target the file management system, memory access, user application, and the C library functions. The motivation and experimental results of each of these primitive benchmarks are presented in detail followed by an analysis of the results. A methodology to combine the primitive benchmarks to form an overall robustness figure is presented. A list of additional primitive benchmarks is suggested.