A Network Load Sensitive Block Placement Strategy of HDFS

• Lingjun Meng,
• Wentao Zhao,
• Haohao Zhao,
• Yang Ding,

Vol. 9, No. 9, September 29, 2015
10.3837/tiis.2015.09.014, Download Paper (Free):

• HDFS
• block placement
• network load
• Imbalance
• Load balance

Abstract

This paper investigates and analyzes the default block placement strategy of HDFS. HDFS is a typical representative distributed file system to stream vast amount of data effectively at high bandwidth to user applications. However, the default HDFS block placement policy assumes that all nodes in the cluster are homogeneous, and places blocks with a simple RoundRobin strategy without considering any nodes｡ｯ resource characteristics, which decreases self-adaptability of the system. The primary contribution of this paper is the proposition of a network load sensitive block placement strategy. We have implemented our algorithm and justify it through extensive simulations and comparison with similar existing studies. The results indicate that our work not only performs much better in the data distribution but also improves write performance more significantly than the others.

Statistics

Cite this article