Fast off-site backup and recovery system for HDFS

HDFS is designed to store massive scale data on commodity hardware reliably. However, it is still vulnerable to severe site disaster, eg. fire, earthquake. The off-site file system backup is an important strategy for data retention. In this paper, we present an efficient, easy-to-use off-site backup and recovery system for HDFS. The system includes a client based on HDFS v2.3.0 with additional feature of off-site backup, and a remote server with a HDFS cluster to keep the backup data. It supports full backup and regularly incremental backup. Both the metadata alteration and the data blocks alteration will be recorded and transferred to remote backup server. Several techniques are used to improve efficiency and throughput. In our experiment, the average speed of backup is up to 95 MB/s, approaching the theoretical maximum speed of gigabit Ethernet.