2023 · 20 citations · 50 references
ComputingComputer EngineeringComputer ScienceDesign OptimizationPerformance MeasurementSoftware EngineeringPerformance Bottleneck AnalysisPerformance ModelingPerformance OptimizationSoftware DesignSoftware ModelingCode CoverageCode Coverage MeasurementPerformance Evaluation ModelsPerformance Prediction ModelsSoftware Performance EngineeringSystem Performance ModelingWorkloadWorkload Characterization
Modern software systems often exhibit numerous configuration options to tailor them to user requirements, including the system's performance behavior. Performance models derived via machine learning are an established approach for estimating and optimizing configuration-dependent software performance. Most existing approaches in this area rely on software performance measurements conducted with a single workload (i.e., input fed to a system). This single workload, however, is often not representative of a software system's real-world application scenarios. Understanding to what extent configuration and workload-individually and combined-cause a software system's performance to vary is key to understand whether performance models are generalizable across different configurations and workloads. Yet, so far, this aspect has not been systematically studied. To fill this gap, we conducted a systematic empirical study across 25 258 configurations from nine real-world configurable software systems to investigate the effects of workload variation at system-level performance and for individual configuration options. We explore driving causes for workload-configuration interactions by enriching performance observations with option-specific code coverage information. Our results demonstrate that workloads can induce substantial performance variation and interact with configuration options, often in non-monotonous ways. This limits not only the generalizability of single-workload models, but also challenges assumptions for existing transfer-learning techniques. As a result, workloads should be considered when building performance prediction models to maintain and improve representativeness and reliability.
50
A Caution Regarding Rules of Thumb for Variance Inflation Factors
Robert M. O’Brien · Quality & Quantity · 2007 · 10.5K citations
A New Measure of Rank Correlation
M. G. Kendall · Biometrika · 1938 · 2.3K citations
Dominance statistics: Ordinal analyses to answer ordinal questions.
Norman Cliff · Psychological Bulletin · 1993 · 1.2K citations
Multicollinearity and Regression Analysis
Jamal Ibrahim Daoud · Journal of Physics Conference Series · 2017 · 1.1K citations · Full text
A Survey on Software Fault Localization
W. Eric Wong, Ruizhi Gao, Yihao Li et al. · IEEE Transactions on Software Engineering · 2016 · 1.1K citations