Evaluating MapReduce for Multi-core and Multiprocessor Systems

TLDR

MapReduce, developed by Google for large‑scale data‑center applications, lets programmers write functional‑style code that is automatically parallelized and scheduled across distributed systems. The paper evaluates the suitability of MapReduce for multi‑core and multi‑processor systems, studying Phoenix’s performance potential and error‑recovery features. Phoenix implements MapReduce for shared‑memory systems, providing a programming API and a runtime that automatically manages thread creation, dynamic task scheduling, data partitioning, fault tolerance, and compares its code to lower‑level P‑threads APIs. The study finds that, with careful implementation, MapReduce is a promising model for scalable performance on shared‑memory systems using simple parallel code.

Abstract

This paper evaluates the suitability of the MapReduce model for multi-core and multi-processor systems. MapReduce was created by Google for application development on data-centers with thousands of servers. It allows programmers to write functional-style code that is automatically parallelized and scheduled in a distributed system. We describe Phoenix, an implementation of MapReduce for shared-memory systems that includes a programming API and an efficient runtime system. The Phoenix runtime automatically manages thread creation, dynamic task scheduling, data partitioning, and fault tolerance across processor nodes. We study Phoenix with multi-core and symmetric multiprocessor systems and evaluate its performance potential and error recovery features. We also compare MapReduce code to code written in lower-level APIs such as P-threads. Overall, we establish that, given a careful implementation, MapReduce is a promising model for scalable performance on shared-memory systems with simple parallel code