Putting Big Data analytics to work: Feature selection for forecasting electricity prices using the LASSO and random forests

Abstract

Successful companies are increasingly those companies that excel in the task of extracting knowledge from data. Tapping the source of ‘Big Data’ requires powerful algorithms combined with a strong understanding of the data used. One of the key challenges in predictive analytics is the identification of relevant factors that may explain the variables of interest. In this paper, we present a case study in predictive analytics in which we focus on the selection of relevant exogenous variables. More specifically, we attempt to predict the German electricity spot prices with reference to historical prices and a deep set of weather variables. In order to choose the relevant weather stations, we use the least absolute shrinkage selection operation (LASSO) and random forests to implicitly execute a variable selection. Overall, in our case study of German weather data, we manage to improve forecasting accuracy by up to 16.9% in terms of mean average error.