An evaluating method of spider detection techniques by trap

Spider is a program for obtaining internet resources. For monitoring spider visits to your website, Decision Tree, Bayesian Network and other Spider Detection Techniques (SDT) are proposed. At present, the evaluation of these detection techniques mainly relies on manual analysis of web log data to calculate the recall rate and precision rate. In order to avoid subjectivity caused by manual analysis, an Evaluation Method based on Trap detection technique of spider (EMT) is proposed in this paper which can evaluate the detecting capability of SDT. The traps layout information on the website and the process information of users accessing website resources are used to calculate relevant parameters, indicators and error range of EMT according to the binomial distribution theory. The Experiment results indicate that EMT and the artificial analysis method have consistent conclusion.