Processing billions of RDF triples on a single machine using streaming and sorting

Abstract

We consider the feasibility of processing billions of RDF triples on a single commodity machine using streaming and sorting techniques and focusing on RDF processing tasks relevant for Linked Data consumption: data filtering and transformation, RDFS inference, owl:sameAs smushing and statistics extraction. To investigate this research question we built RDFpro (rdf processor), an open source tool that provides streaming and sorting-based processors for the considered tasks and allows their sequential and parallel composition in complex pipelines. an empirical evaluation of RDFpro in four application scenario---dataset analysis, filtering, merging and massaging---shows the effectiveness of the tool and allows to positively answer our research question.