Scheduling Job Queue In Hadoop

Computer Science Engineering

Size-based scheduling has been recognized as an effective approach to guarantee fairness and near optimal system response times. HFSP is a scheduler used in this technique to a real, multi-server, complex and widely used system such as Hadoop. It is a widely used bigdata processing engine with a master slave setup. Our solution implements a size-based and pre-emptive scheduling discipline. Scheduling decisions use the concept of virtual time and cluster resource. The jobs are focused according to their priorities that are computed through aging. This ensures that neither small nor large jobs suffer from starvation. The Size based scheduling in HFSP, gives the priority to small jobs that they will not be slowed down by large ones. The Shortest Remaining Processing Time (SRPT) concept gives priority to the jobs that requires minimum amount of work to complete. It minimizes the mean response time (or sojourn time), that is the time that passes between a job submission and its completion. We Extend HFSP to pause jobs with Higher SRPT and allow other waiting jobs in Queue based on FCFS.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.