Job-aware Network Scheduling for Hadoop Cluster

• Wen Liu,
• Zhigang Wang,
• Yanming Shen,

Vol. 11, No. 1, January 29, 2017
10.3837/tiis.2017.01.012, Download Paper (Free):

• Cloud computing
• job-aware
• priority scheduling
• Hadoop
• load balancing

Abstract

In recent years, data centers have become the core infrastructure to deal with big data processing. For these big data applications, network transmission has become one of the most important factors affecting the performance. In order to improve network utilization and reduce job completion time, in this paper, by real-time monitoring from the application layer, we propose job-aware priority scheduling. Our approach takes the correlations of flows in the same job into account, and flows in the same job are assigned the same priority. Therefore, we expect that flows in the same job finish their transmissions at about the same time, avoiding lagging flows. To achieve load balancing, two approaches (Flow-based and Spray) using ECMP (Equal-Cost multi-path routing) are presented. We implemented our scheme using NS-2 simulator. In our evaluations, we emulate real network environment by setting background traffic, scheduling delay and link failures. The experimental results show that our approach can enhance the Hadoop job execution efficiency of the shuffle stage, significantly reduce the network transmission time of the highest priority job.

Statistics

Cite this article