Big Picture of Big Data Software Engineering: With Example Research Challenges

Abstract

In the rapidly growing field of Big Data, we note that a disproportionately larger amount of effort is being invested in infrastructure development and data analytics in comparison to applications software development -- approximately a 80:20 ratio. This prompted us to create a context model of Big Data Software Engineering (BDSE) containing various elements -- such as development practice, Big Data systems, corporate decision-making, and research -- and their relationships. The model puts into perspective where various types of stakeholders fit in. From the research perspective, we describe example challenges in BDSE, specifically requirements, architectures, and testing and maintenance.