GENETIC ALGORITHM FOR EVALUATION METRICS IN TOPICAL WEB CRAWLING

A topic driven crawler chooses the best URLs to pursue during web crawling. It is difficult to evaluate what URLs downloaded are the best. This paper presents some important metrics and an evaluation function for ranking URLs about pages relevance. We also discuss an approach to evaluate the function based on GA. The best combination of the metrics' weights can be discovered by GA evolving process. The experiment shows that the performance is exciting, especially about a popular topic.