A Study of Heuristically-Based Parametric Performance Improvement/Optimization Algorithms for BigData Computing

Performance optimization for mapreduce computing in Hadoop platform is a tedious yet challenging problem due to the complexity of system organization with an extensive list of configuration parameters to be considered. In order to address and resolve this problem, various parameter optimization algorithms are proposed in this research from a naive exhaustive method to a random and a couple of heuristically-based greedy methods to vie with the exponentially nature of the search process for the possible best parameter setting. Extensive benchmark-based experiments have been conducted to validate the performance viability of the mapreduce computations by the benchmark programs such as TestDFSIO, TeraSort, to mention a couple. The experimental results demonstrate the proposed heuristically-based algorithms in greedy manner provide a promising answer to the problem of the research how to optimize the systems configuration parameter set at a computationally viable and feasible cost.