Data Scientists in Software Teams: State of the Art and Challenges

TLDR

Demand for large‑scale telemetry, machine, and quality data analysis is rising, and data scientists are increasingly integrated into software teams at companies such as Facebook, LinkedIn, and Microsoft. This paper surveys 793 Microsoft data scientists to characterize their educational backgrounds, problem domains, tool usage, and activities. The authors cluster participants by time spent on activities, identifying nine distinct groups, and discuss the challenges they face and best practices shared among them. The survey uncovers trends in Microsoft data scientists’ profiles and work, providing managers with insights to better leverage data science capabilities within teams.

Abstract

The demand for analyzing large scale telemetry, machine, and quality data is rapidly increasing in software industry. Data scientists are becoming popular within software teams, e.g., Facebook, LinkedIn and Microsoft are creating a new career path for data scientists. In this paper, we present a large-scale survey with 793 professional data scientists at Microsoft to understand their educational background, problem topics that they work on, tool usages, and activities. We cluster these data scientists based on the time spent for various activities and identify 9 distinct clusters of data scientists, and their corresponding characteristics. We also discuss the challenges that they face and the best practices they share with other data scientists. Our study finds several trends about data scientists in the software engineering context at Microsoft, and should inform managers on how to leverage data science capability effectively within their teams.