Prediction of research trends using LDA based topic modeling

Abstract

Change is the only constant. In many sectors, a change is being witnessed that is getting increasingly rapid. This carries a plethora of new innovation possibilities with it. This necessitates well-founded data about trends, future developments and their consequences. This study seeks to catch the new directions, paradigms as predictors with an association of each topic which will be discovered through topic modeling techniques like LDA with BoW. For this, empirical analysis on 3269 research articles from the Journal of Applied Intelligence was done, which were gathered during a 30-year span. The inferred topics were then structured into a way suitable for performing predictive analysis. This is significant in the sense that it will help to predict what technology will be encountered in the future, as well as how far human's ability to innovate and discover things may lead this world to. The final model using TF-IDF scores has outperformed the baseline model by a margin of 41%.