Detecting Android Security Vulnerabilities Using Machine Learning and System Calls Analysis

Android operating systems have become a prime target for cyber attackers due to security vulnerabilities in the underlying operating system and application design. Recently, anomaly detection techniques are widely studied for security vulnerabilities detection and classification. However, the ability of the attackers to create new variants of existing malware using various masking techniques makes it harder to deploy these techniques effectively. In this research, we present a robust and effective vulnerabilities detection approach based on anomaly detection in a system calls of benign and malicious Android application. The anomaly in our study is type, frequency, and sequence of system calls that represent a vulnerability. Our system monitors the processes of benign and malicious application and detects security vulnerabilities based on the combination of parameters and metrics, i.e., type, frequency and sequence of system calls to classify the process behavior as benign or malign. The detection algorithm detects the anomaly based on the defined scoring function f and threshold ρ. The system refines the detection process by applying machine learning techniques to find a combination of system call metrics and explore the relationship between security bugs and the pattern of system calls detected. The experiment results show the detection rate of the proposed algorithm based on precision, recall, and f-score for different machine learning algorithms.