MBTree: Detecting Encryption RAT Communication Using Malicious Behavior Tree

A key challenge for cybersecurity defense is to detect the encryption Remote Control Trojan (RAT) communication traces. It is still an open research problem to detect encryption RAT preciously in different environments. Previous studies in this area either cannot handle the encrypted content or perform unstable in a different environment. To tackle both problems, we present MBTree, a novel host-level signature based approach for encryption RAT traffic detection. MBTree consists of a structure named MLTree and a similarity matching mechanism. The MLTree integrates multiple directed packet payload size sequences as a host signature. Furthermore, the matching mechanism compares two MLTree to decide if an alarm is triggered. Compared with previous related studies, MBTree (i) is more accurate to characterize different encryption RATs; (ii) has more robust performance when emerging new benign applications in the test environment; (iii) can automatically create signatures from malicious traffic without requiring human interaction. For evaluation, we collect traffic from multiple sources and reorganize them in a sophisticated manner. The experiment results demonstrate that our proposed method is more precise and robust, especially in the situation with new emerging applications.