Analyzing outliers cautiously

Abstract

Outliers are difficult to handle because some of them can be measurement errors, while others may represent phenomena of interest, something "significant" from the viewpoint of the application domain. Statistical and computational methods have been proposed to detect outliers, but further analysis of outliers requires much relevant domain knowledge. In our previous work (1994), we suggested a knowledge-based method for distinguishing between the measurement errors and phenomena of interest by modeling "real measurements" - how measurements should be distributed in an application domain. In this paper, we make this distinction by modeling measurement errors instead. This is a cautious approach to outlier analysis, which has been successfully applied to a medical problem and may find interesting applications in other domains such as science, engineering, finance, and economics.