The emerging role of data scientists on software development teams

TLDR

Software development generates large amounts of raw data that can be turned into actionable insight with skilled data scientists, but such professionals have been scarce until recent efforts by software companies to build software‑oriented data analytics competencies. The paper describes data scientists’ education and training, their missions in software engineering contexts, and the types of problems they tackle. The authors interviewed data scientists from multiple Microsoft product groups and outlined the strategies they use to enhance the impact and actionability of their work. The study identifies five distinct data scientist working styles—Insight Providers, Modeling Specialists, Platform Builders, Polymaths, and Team Leaders—each with specific roles in software engineering.

Abstract

Creating and running software produces large amounts of raw data about the development process and the customer usage, which can be turned into actionable insight with the help of skilled data scientists. Unfortunately, data scientists with the analytical and software engineering skills to analyze these large data sets have been hard to come by; only recently have software companies started to develop competencies in software-oriented data analytics. To understand this emerging role, we interviewed data scientists across several product groups at Microsoft. In this paper, we describe their education and training background, their missions in software engineering contexts, and the type of problems on which they work. We identify five distinct working styles of data scientists: (1) Insight Providers, who work with engineers to collect the data needed to inform decisions that managers make; (2) Modeling Specialists, who use their machine learning expertise to build predictive models; (3) Platform Builders, who create data platforms, balancing both engineering and data analysis concerns; (4) Polymaths, who do all data science activities themselves; and (5) Team Leaders, who run teams of data scientists and spread best practices. We further describe a set of strategies that they employ to increase the impact and actionability of their work.