An evaluation of Twitter river and Logstash performances as elasticsearch inputs for social media analysis of Twitter

Abstract

Social media analysis of Twitter can be used to show a rating of someone, a service, or a product from Twitter user's perspective. As one of social media with the highest number of users in the world, Twitter provides an API that allows us to observe and take Twitter data in real-time. Elasticsearch is a tool that has the ability to analyze big data. There are two ways to input Twitter data to Elasticsearch. The first one is through Twitter River and the second way is through Logstash. This input factor is important in influencing the output of the system. Accuracy and efficiency of input data and the way of data is stored is really important to support a system of big data. In this paper, an evaluation of Twitter River and Logstash performances as in case of inputting Twitter data from Twitter API is presented. This research monitors Elasticsearch cluster on two HPC servers that crawls data from Twitter API simultaneously. Comparing parameters are CPU process, RAM usage, disk usage, Twitter input data, and amount of input fields. The result of this research shows that the average CPU process per day of Twitter River is 33.96%, and for Logstash 34.95%. The average RAM usage of Twitter River per day is 32.7% while Logstash used 39.9%. Besides, the average disk usage of Twitter River per day is 431 MB and for Logstash 544 MB. For the Twitter input data, Twitter River inputs 191 more tweet than Logstash in a week. And the result shows that Logstash inputting 11 times field more than Twitter River.