The big data ecosystem at LinkedIn

Roshan Sumbaly, Jay Kreps, Sam Shah

2013 · 121 citations · 31 references

Concepts

TL;DR

Large‑scale data mining and machine learning have proliferated with Hadoop’s simple semantics and active ecosystem. The paper presents LinkedIn’s Hadoop‑based analytics stack and solutions to last‑mile challenges, enabling data scientists to extract insights and build product features from massive data. The stack offers easy ingress and egress to online systems and manages workflows as production processes, built on Hadoop. The solution abstracts distributed system concerns, allowing researchers to deploy data with a single Pig command, and case studies demonstrate its use in recommendations, news‑feed updates, email digesting, and dashboards.

Abstract

The use of large-scale data mining and machine learning has proliferated through the adoption of technologies such as Hadoop, with its simple programming semantics and rich and active ecosystem. This paper presents LinkedIn's Hadoop-based analytics stack, which allows data scientists and machine learning researchers to extract insights and build product features from massive amounts of data. In particular, we present our solutions to the ``last mile'' issues in providing a rich developer ecosystem. This includes easy ingress from and egress to online systems, and managing workflows as production processes. A key characteristic of our solution is that these distributed system concerns are completely abstracted away from researchers. For example, deploying data back into the online system is simply a 1-line Pig command that a data scientist can add to the end of their script. We also present case studies on how this ecosystem is used to solve problems ranging from recommendations to news feed updates to email digesting to descriptive analytical dashboards for our members.

References

31