• KSII Transactions on Internet and Information Systems
    Monthly Online Journal (eISSN: 1976-7277)

Job-aware Network Scheduling for Hadoop Cluster

Vol. 11, No. 1, January 29, 2017
10.3837/tiis.2017.01.012, Download Paper (Free):

Abstract

In recent years, data centers have become the core infrastructure to deal with big data processing. For these big data applications, network transmission has become one of the most important factors affecting the performance. In order to improve network utilization and reduce job completion time, in this paper, by real-time monitoring from the application layer, we propose job-aware priority scheduling. Our approach takes the correlations of flows in the same job into account, and flows in the same job are assigned the same priority. Therefore, we expect that flows in the same job finish their transmissions at about the same time, avoiding lagging flows. To achieve load balancing, two approaches (Flow-based and Spray) using ECMP (Equal-Cost multi-path routing) are presented. We implemented our scheme using NS-2 simulator. In our evaluations, we emulate real network environment by setting background traffic, scheduling delay and link failures. The experimental results show that our approach can enhance the Hadoop job execution efficiency of the shuffle stage, significantly reduce the network transmission time of the highest priority job.


Statistics

Show / Hide Statistics

Statistics (Cumulative Counts from December 1st, 2015)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article

[IEEE Style]
W. Liu, Z. Wang and Y. Shen, "Job-aware Network Scheduling for Hadoop Cluster," KSII Transactions on Internet and Information Systems, vol. 11, no. 1, pp. 237-252, 2017. DOI: 10.3837/tiis.2017.01.012.

[ACM Style]
Wen Liu, Zhigang Wang, and Yanming Shen. 2017. Job-aware Network Scheduling for Hadoop Cluster. KSII Transactions on Internet and Information Systems, 11, 1, (2017), 237-252. DOI: 10.3837/tiis.2017.01.012.