An experimental study on execution time variation in computer experiments

Abstract

In computer experiments, many research works rely on the accuracy of measured programs' execution time. We observe that not all studies consider that repeated executions of the same program, under the same experimental conditions, may produce statistically significant different completion times. In this work, we experimentally demonstrate that several sources of OS Jitter affect the execution time of computer programs. We compare various execution time samples using three test protocols, which apply different statistical techniques. The results show that significant differences are detected in all evaluated scenarios.