Using Big Data and Real-Time Analytics to Support Smart City Initiatives

TLDR

Smart city initiatives require timely acquisition of high‑throughput, large‑volume, often ill‑formed natural‑language data from sources such as Twitter. The paper presents a platform for processing Twitter messages to support smart city analytics. The platform employs pipelined modules for real‑time tweet acquisition, storage, filtering, NLP, sentiment analysis, and visualization. A 2014 FIFA World Cup Brazil sentiment‑analysis case study demonstrates the platform’s effectiveness.

Abstract

A central issue in the context of smart cities is related to the capability to acquire timely information about city events. This paper describes a platform which focuses on processing messages posted in Twitter social network. Key issues here are the high throughput a large volume of data per second that needs to be processed, and the need to process ill formed natural language texts. With these in mind the platform has pipelined modules for robust, fast, real time tweet acquisition and storage, filtering of several kinds, natural language processing and sentiment analysis, that feed a final analysis and visualization module. A case study of sentiment analysis during the 2014 FIFA World Cup in Brazil is used to validate the effort made so far.