"EXMR; An Enhanced way for Incremental Data Processing based on MapReduce"

Computer Engineering

Map Reduce programming model is widely used for large scale and one-time data-intensive distributed computing, but lacks flexibility and efficiency of processing small incremental data. Incremental Mapreduce framework is proposed in this paper for incrementally processing new data of a large data set, which takes state as implicit input and combines it with new data. Map tasks are created according to new splits instead of entire splits while reduce tasks fetch their inputs including the state and the intermediate results of new map tasks from Preserved and latest generated result. The preserved states really producing the promising result and significantly reduce the run time for refreshing big data mining results compared to re-calculating on both simple and multi stage MapReduce. The intermediate states are saved in the form of kv-pair level data flow and data dependence in a MapReduce computation as a bipartitegraph, called MRBGraph. A MRBG-Store is designed to preserve the fine-grain states in the MRBGraph and support efficient queries to retrieve fine-grain states for incremental processing.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.