Beyond the Hype: Why Do Data-Driven Projects Fail?

TLDR

Despite substantial investments, data science has failed to deliver significant business value in many companies, and the reasons for this problem have not been systematically explored. This study seeks to identify explanations for the shortfall and analyze specific challenges in data‑driven projects. The authors conducted multiple rounds of qualitative semi‑structured interviews with domain experts across roles, followed by a questionnaire of 112 experts from eleven industries. The study finds that failures stem mainly from lack of business context understanding, low data quality, and data access problems, with 54 % of respondents noting a conceptual gap between strategy and analytics implementation, and offers recommendations to bridge this gap.

Abstract

Despite substantial investments, data science has failed to deliver significant business value in many companies. So far, the reasons for this problem have not been explored systematically. This study tries to find possible explanations for this shortcoming and analyses the specific challenges in data-driven projects. To identify the reasons that make data-driven projects fall short of expectations, multiple rounds of qualitative semi-structured interviews with domain experts with different roles in data-driven projects were carried out. This was followed by a questionnaire surveying 112 experts with experience in data projects from eleven industries. Our results show that the main reasons for failure in data-driven projects are (1) the lack of understanding of the business context and user needs, (2) low data quality, and (3) data access problems. It is interesting, that 54% of respondents see a conceptual gap between business strategies and the implementation of analytics solutions. Based on our results, we give recommendations for how to overcome this conceptual distance and carrying out data-driven projects more successfully in the future.