Comparative Evaluation of Architectural and Code-Level Approaches for Finding Security Vulnerabilities

During architectural risk analysis, Security Information Workers (SIWs) reason about security-relevant architectural flaws using a high-level representation of the system's structure instead of directly reading the code as in during a code review. It is still hard to extract from the code a high-level representation that is sound, conveys design intent, and enables expressive constraints that can find security vulnerabilities. As a result, architecture-level approaches are less mature than code-level ones that extract low-level representations that are not directly intended for use by SIWs. In this paper, we compare an architecture-level approach with a code-level approach in terms of effectiveness (precision and recall) across test cases with injected vulnerabilities that range from coding bugs to architectural flaws. The evaluation shows that an architecture-level approach can uncover some security vulnerabilities with better precision and recall than a code-level approach. Moreover, it shows that the effectiveness of the approaches varies greatly based on whether the security vulnerability is a coding bug or an architectural flaw. These results may help SIWs select the right tools for the job of securing their systems.