Performance Enhancement of Hadoop for Big Data Using Multilevel Queue Migration (MQM) Technique

The recent advancements in Hadoop MapReduce scheduling techniques have demonstrated significant outcomes. The continuous tradeoff between the data-job locality and synchronization results in the higher efficiency for the framework. Thus, large number of scientific and enterprise applications have adopted the parallel and synchronized mechanism through Hadoop framework. However, with this adaptation, a large number of datacenter-based nodes are been deployed, significantly causing the increase of energy consumptions. Henceforth, the demand of the recent research is to enhance the overall efficiency of Hadoop jobs and to decrease the energy consumption without degrading the performance. The recent advancements have demonstrated by many strategies by improving the Map and Reduce job allocation techniques; conversely, the same improvement can also be achieved through multilevel queues. Hence, this work constitutes the multilevel queue with custom load balancing to demonstrate the improvement in overall performance of Hadoop job scheduling. The work results in a significant improvement of Hadoop jobs in terms of execution times and energy consumption.