Reducing failure analysis time: an industrial evaluation

Abstract

Testing and debugging automotive cyber physical systems are challenging. Developing and integrating cyber and physical components require extensive testing to ensure reliable and safe releases. One important cost factor in the debugging process is the time required to analyze failures. Since large number of failures usually happen due to a few underlying faults, clustering failures based on the responsible faults helps reduce analysis time. We focus on the software-in-the-loop and hardware-in-the-loop levels of testing where test execution times are high. We devise a methodology for adapting existing clustering techniques to a real context. We augment an existing clustering approach by a method for selecting representative tests. To analyze failures, rather than investigating all failing tests one by one, testers inspect only these representatives. We report on the results of a large scale industrial case study. We ran experiments on ca. 850 KLOC. Results show that utilizing our clustering tool, testers can reduce failure analysis time by more than 80%.