Big Data: Issues and Challenges Moving Forward

TLDR

Big data encompasses exabyte‑scale volumes that exceed current storage and processing capacities, rapidly approaching zettabyte/year rates, and is characterized by additional attributes of variety, velocity, value, and complexity, posing near‑term solvable storage and transport challenges that will require long‑term research and new paradigms. The study aims to analyze the issues and challenges of big data as part of a collaborative research program developing new methodologies for its analysis and design. The authors initiate a collaborative research program that analyzes big data issues and challenges to develop new analytical and design methodologies.

Abstract

Big data refers to data volumes in the range of exabytes (1018) and beyond. Such volumes exceed the capacity of current on-line storage systems and processing systems. Data, information, and knowledge are being created and collected at a rate that is rapidly approaching the exabyte/year range. But, its creation and aggregation are accelerating and will approach the zettabyte/year range within a few years. Volume is only one aspect of big data; other attributes are variety, velocity, value, and complexity. Storage and data transport are technology issues, which seem to be solvable in the near-term, but represent longterm challenges that require research and new paradigms. We analyze the issues and challenges as we begin a collaborative research program into methodologies for big data analysis and design.