A Mathematical Modeling of Exploitations and Mitigation Techniques Using Set Theory

One of the most challenging problems in computer security is formalization of vulnerabilities, exploits, mitigations and their relationship. In spite of various existing researches and theories, a mathematical model that can be used to quantitatively represent and analyze exploit complexity and mitigation effectiveness is still in absence. In this work, we introduce a novel way of modeling exploits and mitigation techniques with mathematical concepts from set theory and big O notation. The proposed model establishes formulaic relationships between exploit primitives and exploit objectives, and enables the quantitative evaluation of vulnerabilities and security features in a system. We demonstrate the application of this model with two real world mitigation techniques. It serves as the first step toward a comprehensive mathematical understanding and modeling of exploitations and mitigations, which will largely benefit and facilitate the practice of system security assessment.