Improving execution speed of incremental runs of MapReduce using provenance

Hadoop MapReduce is an analytic tool used to solve big data problems that are parallelisable. MapReduce jobs need to be rerun frequently for data changes. Many times these data changes are made by appending new data to the existing file. So in a rerun, if we can reuse the output of the previous run and limit the job execution to the new data, we can reduce the overall job execution time. In the literature, there are schemes that use the concept of memoisation, storing the intermediate result, etc. to implement efficient incremental rerun. In this paper, we explain how provenance can be used to implement transparent incremental MapReduce for 'append only' input files. Our approach requires no additional storage or modification of existing Hadoop infrastructure or scheduler. Experimental evaluation of running MapReduce on multinode cluster with provenance stored in HBase gave good results for incremental runs in cases of an addition of new file/new data.