CHIEv: concurrent hybrid analysis for crawling and modeling of web applications

Researchers and practitioners in the fields of testing, security assessment and web development seeking to evaluate a given web application often have to rely on the existence of a model of the respective system, which is then used as input to task-specific tools. Such models may include information on HTTP endpoints and their parameters, available user actions/event listeners and required assets. Unfortunately, this data is often unavailable in practice, as only rigorous development practices or manual analysis guarantee their existence and correctness. Crawlers based on static analysis have traditionally been used to extract required information from existing sites. Regrettably, these tools can not accurately account for the dynamic behavior introduced by technologies such as JavaScript that are prevalent on modern sites. While methods based on dynamic analysis exist, they are often not fully capable of identifying event listeners and their effects. In an earlier work, we presented XIEv, an approach for dynamic analysis of web applications that produces an execution trace usable for the extraction of navigation graphs, identification of bugs at runtime and enumeration of resources. It offers improved recognition and selection of event listeners as well as a greater range of observed effects compared to existing approaches. While the evaluation of our research prototype implementation confirmed the capabilities of XIEv, it was generally out-performed by static crawlers in terms of speed. This work introduces CHIEv, an approach that augments XIEv by enabling concurrent processing as well as incorporating the results of a static crawler in real-time. Our results indicate a significant increase in performance, particularly when applied to larger sites.