DLME: Distributed Log Mining Using Ensemble Learning for Fault Prediction

Abstract

Fault prediction problems in network systems are often manifested as very onerous for better network management. One of the effective measures is to constantly monitor and analyze the unceasing generation of network logs that capture the activities of a network. The learning algorithms are quite useful for this purpose. However, due to the dynamic nature of network systems, a frequent drift in the logged data may occur which in turn affects the efficiency of the learning algorithms. In this paper, we present a general purpose algorithmic framework for developing easily parallelizable distributed log mining approach, which uses machine learning and distributed processing to achieve a better quality of network services. Our proposed approach monotonously handles the dynamic nature of network logs by tracking the changes in the distribution of logs and takes adequate actions according to that. The entire problem is illustrated as a distributed learning environment, where the complete set of logs is partitioned into assorted data chunks and a distributed weighted ensemble of the information is generated from these chunks. Furthermore, our method is tested on real dataset and experimental analysis shows that a fair amount of scalability and accuracy can be obtained.