Stupid Data Miner Tricks: Overfitting the S&P 500

Abstract

This article originated over ten years ago as a set of joke slides showing silly spurious correlations. These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often, in Business Week, the Wall Street Journal, the book “A Mathematician Looks at the Stock Market,” and elsewhere. Students from Bill Sharpe9s classes at Stanford seem to be familiar with them. The slides were expanded to include some actual content about data mining, and reissued as an academic working paper in 2001. Occasional requests arrive from distant corners of the world, so I thank the editors of the Journal of Investing for publishing this article. Without taking a hatchet to the original, the advice offered remains valuable, perhaps even more so now that there is so much more data to mine. Monthly data arrives as a single data point, once a month. It9s hard to avoid data mining sins if you look twice. Ticks, quotes, and executions arrive in millions per minute, and many of the practices which fail the statistical sniff tests for low frequency data can now be used responsibly. Nevertheless, fooling yourself remains an occupational hazard in quantitative trading.