A hybrid approach to forecasting futures prices with simultaneous consideration of optimality in ensemble feature selection and advanced artificial intelligence

Abstract

The paper presents a framework to forecast futures prices of stocks listed on the National Stock Exchange (NSE) in India during normal (unaffected by the COVID-19 pandemic) and new normal times (affected by COVID-19 and a macroeconomic slowdown). The model leverages a structural model that determines the relevance of the explanatory features used in the study; namely, spot prices, market sentiment, sectoral outlook, historic and implied volatility, crude price volatility, and exchange rate volatility. The proposed Ensemble Feature Selection (EFS) methodology comprising Boruta and Regularized Random Forest (RRF) algorithms is used to screen the explanatory features. Two advanced Artificial Intelligence techniques—Regularized Greedy Forest (RGF) and Deep Neural Network (DNN)—are used in conjunction with Kernel Principal Component Analysis (KPCA) and Autoencoder (AE) for forecasting. To understand the extent and nature of the influence of the explanatory variables, the Explainable Artificial Intelligence (AI) approach has been used. Statistical checks confirm that our hybrid framework is effective. The results indicate that the relative importance of the explanatory variables in forecasting futures prices differs depending on the company concerned and the period under consideration.