Shellfier: A Shellcode Detection Method based on Dynamic Binary Instrumentation and Convolutional Neural Network

An important method of detecting zero-day attacks is to identify the shellcode which is usually taken as part of the attacks. It is vital to detect programs that have the characteristics of shellcode behavior in the network traffic detection. In this paper, a shellcode detection method named Shellfier based on Dynamic Binary Instrumentation and Convolutional Neural Network (CNN) is proposed. The method of program instrumentation can obtain the behavior characteristics of shellcode in fine-grained manner. The CNN algorithm trains and classifies the sample data, and compares the classification effect of Support Vector Machine (SVM) algorithm based on n-grams model to extract feature vectors. The experimental results show that CNN has strong representation ability for behavioral characteristics, which is more accurate than SVM classification, and the false positive rate and vulnerability rate are lower.