A New Improved Baum-Welch Algorithm for Unsupervised Learning for Continuous-Time HMM Using Spark

Abstract

Hidden Markov Models are widely used for time continuous problems modelling and prediction. This paper presents two new improved algorithms for Gaussian continuous and mixture of Gaussian continuous Hidden Markov Models cases for solving learning problem for large scale multidimensional data. The design of our parallel distributed algorithms is based on Spark, the Big Data framework, thereby we can distribute data over several nodes through Resilient Distributed Datasets which allow to apply, in parallel, a set of operations. The proposed algorithms have two main advantages: a high computational time efficiency and a high scalability as well as an easy integration in Big Data frameworks. The complexity comparison results show great improvements in computational complexity (by a factor of (states number) 2 ) and execution time. Moreover, the new algorithms might be more effective by reducing the communication costs between the elements of the system involved in the learning task.