Data quality in big data processing: Issues, solutions and open problems

Abstract

With the rapid development of social networks, Internet of things, Cloud computing as well as other technologies, big data age is arriving. The increasing number of data has brought great value to the public and enterprises. Meanwhile how to manage and use big data better has become the focus of all walks of life. The 4V characteristics of big data have brought a lot of issues to the big data processing. The key to big data processing is to solve data quality issue, and to ensure data quality is a prerequisite for the successful application of big data technique. In this paper, we use recommendation systems and prediction systems as typical big data applications, and try to find out the data quality issues during data collection, data preprocessing, data storage and data analysis stages of big data processing. According to the elaboration and analysis of the proposed issues, the corresponding solutions are also put forward. Finally, some open problems to be solved in the future are also raised.