ComputingComputer ScienceArtificial IntelligenceFeature EngineeringMachine LearningDeep LearningFeature ConstructionFeature TransformationNeural NetworksRepresentation LearningScalable Machine LearningConvolutional Neural Network SegmentationConvolutional Neural NetworksData ProcessingDeep Feature LearningFeature LearningKernel LearningPrototypingRapid Experimentation
Graph algorithm computations are data‑driven and demand high random data access, yet disk technology cannot provide efficient random access and memory‑based approaches lack scalability due to capacity limits. The paper introduces Trinity, a general‑purpose graph engine that runs on a distributed memory cloud. Trinity achieves fast graph exploration and efficient parallel computing through optimized memory management, network communication, and a high‑level specification language (TSL) that adapts to graph access patterns in both online and offline modes. Experiments demonstrate that Trinity delivers low‑latency graph queries and high‑throughput analytics on web‑scale, billion‑node graphs using only a few commodity machines.
Computations performed by graph algorithms are data driven, and require a high degree of random data access. Despite the great progresses made in disk technology, it still cannot provide the level of efficient random access required by graph computation. On the other hand, memory-based approaches usually do not scale due to the capacity limit of single machines. In this paper, we introduce Trinity, a general purpose graph engine over a distributed memory cloud. Through optimized memory management and network communication, Trinity supports fast graph exploration as well as efficient parallel computing. In particular, Trinity leverages graph access patterns in both online and offline computation to optimize memory and communication for best performance. These enable Trinity to support efficient online query processing and offline analytics on large graphs with just a few commodity machines. Furthermore, Trinity provides a high level specification language called TSL for users to declare data schema and communication protocols, which brings great ease-of-use for general purpose graph management and computing. Our experiments show Trinity's performance in both low latency graph queries as well as high throughput graph analytics on web-scale, billion-node graphs.
28
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
Spark: cluster computing with working sets
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin et al. · 2010 · 4.2K citations
Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik et al. · 2010 · 3.5K citations
PowerGraph: distributed graph-parallel computation on natural graphs
Joseph E. Gonzalez, Yucheng Low, Haijie Gu et al. · 2012 · 1.5K citations