The design and benchmarking of a Cloud-based platform for processing and visualization of traffic data

Abstract

Nowadays, more and more data are being generated and collected. This is especially so in our cities. Big data as a popular research topic has brought about a range of new challenges that must to be tackled to support both commercial and research demands. The transport area is one example that has much to benefit from big data capabilities in allowing to process voluminous amounts of data that is created in real time and in vast quantities. Tackling these big data issues requires capabilities not typically found in common Cloud platforms. The high speed of production of data (velocity) of high volume data demands a distributed file system for storing and duplicating data; high performance computing engines capable of processing such large quantities of data in real time; a reliable database system able to optimize the indexing and querying of the data, and essentially a flexible implementation for scaling the usage of computation resources based on the bursty nature of data production. In this paper, we present and benchmark SMASH, a generic and highly scalable Cloud-based solution. We focus here on the implementation and especially on the utilization of the SMASH software stack and how it can be used to process large scale traffic data - although we note that the solution can be applied to other high velocity big data processing areas.