MassStore: A low bandwidth, high De-duplication efficiency network backup system

De-duplication technology has been widely used in disk-based backup system in order to save disk space and reduce backup traffic through internet. But unfortunately De-duplication based backup system often has metadata indexing bottleneck that greatly reduces the backup efficiency and throughput. Existing approaches usually take advantage of backup data flow's similarity or locality to accelerate metadata indexing. In this paper, we design and implement MassStore, a de-duplication based network backup system which use a two-stage locality sensitive hash algorithm, that combines backup data flow's data similarity within data flow's chunk set and the locality between different chunk sets, to accelerate metadata indexing so as to improve de-duplication efficiency. The experimental results using real word data sets shows that our MassStore not only saved the backup storage by average of 88.5%, but also reduced the network bandwidth and RAM usage.