2013 · 121 citations · 31 references
ComputingData ScienceBig Data AnalyticsData ManagementData MiningData IntegrationData OrchestrationData PipelinesDistributed ComputingDistributed Data ProcessingMachine Learning For Big DataAnalytical DashboardsBig Data ArchitectureDashboardsDistributed Machine LearningLarge-scale Data ProcessingMachine Learning AnalyticsProduct FeaturesScalable Data Management
Large‑scale data mining and machine learning have proliferated with Hadoop’s simple semantics and active ecosystem. The paper presents LinkedIn’s Hadoop‑based analytics stack and solutions to last‑mile challenges, enabling data scientists to extract insights and build product features from massive data. The stack offers easy ingress and egress to online systems and manages workflows as production processes, built on Hadoop. The solution abstracts distributed system concerns, allowing researchers to deploy data with a single Pig command, and case studies demonstrate its use in recommendations, news‑feed updates, email digesting, and dashboards.
The use of large-scale data mining and machine learning has proliferated through the adoption of technologies such as Hadoop, with its simple programming semantics and rich and active ecosystem. This paper presents LinkedIn's Hadoop-based analytics stack, which allows data scientists and machine learning researchers to extract insights and build product features from massive amounts of data. In particular, we present our solutions to the ``last mile'' issues in providing a rich developer ecosystem. This includes easy ingress from and egress to online systems, and managing workflows as production processes. A key characteristic of our solution is that these distributed system concerns are completely abstracted away from researchers. For example, deploying data back into the online system is simply a 1-line Pig command that a data scientist can add to the end of their script. We also present case studies on how this ecosystem is used to solve problems ranging from recommendations to news feed updates to email digesting to descriptive analytical dashboards for our members.
31
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Mining association rules between sets of items in large databases
Rakesh Agrawal, Tomasz Imieliński, Arun Swami · 1993 · 14.8K citations · Full text
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani et al. · 2007 · 3.5K citations