Rapid data de-duplication method adapted to big data application

The invention provides a rapid data de-duplication method adapted to a big data application. The rapid data de-duplication method is applied to backup de-duplication systems under the big data application and solves the problems that the existing variable-length partition algorithm based on content identification is low in de-duplication rate and fails to identify redundant data rapidly. According to the rapid data de-duplication method, through adjusting de-duplication factors and acceleration factors in a partition process, the de-duplication rate is substantially improved on the premise that the de-duplication ratio is ensured, de-duplication detection can be performed rapidly, the contradiction between the de-duplication ratio and the de-duplication rate is balanced, backup windows are reduced, and network bandwidth and memory spaces are saved.