Design of analysis system for documents based on web crawler

This paper studies the technologies of information extraction and data mining. By using web crawlers to analyze and process the specified block of text, a system for document analysis is completed. For specific needs, this system is able to simulate the working process of the search engines. By intercepting the data flow, the automatic extraction and filtering of the document are achieved, so it effectively improves the automation of document information extraction.