Research on an anti-crawling mechanism and key algorithm based on sliding time window

Inadequate crawling behavior of the crawler will have a very serious impact on the site, so anti crawling mechanism is an important function for the website. Most of the existing anti crawling methods are non real time detection, and the recognition accuracy is low. By analyzing the characteristics of Crawler, a real-time crawler detection method based on sliding time window is proposed, which improves the accuracy and efficiency of detection of non compliance with the rules of Crawler.