Stocks Analysis and Prediction Using Big Data Analytics

Abstract

Big data analytics are used primarily in various sectors for accurate prediction and analysis of the large data sets. They allow the discovery of significant information from large data sets, otherwise, it is hidden. In this paper, an approach of robust Cloudera-Hadoop based data pipeline is proposed to perform analyses for any scale and type of data, in which selected US stocks are analysed to predict daily gains based on real time data from Yahoo Finance. The Apache Hadoop big-data framework is provided to handle large data sets through distributed storage and processing, stocks from the US stock market are picked and their daily gain data are divided into training and test data set to predict the stocks with high daily gains using Machine Learning module of Spark.