Inspector: A Semantics-Driven Approach to Automatic Protocol Reverse Engineering

The automatic protocol reverse engineering for undocumented network protocols is important for many fundamental network security applications, such as intrusion prevention and detection. With the growing prevalence of binary protocols in the network communication to organize data in a terse format and ensure data integrity, the proven reverse approaches for ordinary text protocols face severe challenges in the compatibility. In this paper, we propose Inspector, an automatic protocol reverse engineering approach that exploits semantic fields to infer message formats from binary network traces. Inspector reasonably infers two semantic fields based on the binary content analysis of protocol messages to support clustering messages and message format inference. We evaluate the effectiveness of Inspector on two binary cryptographic protocols (TLS and SSH) and a binary unencrypted protocol MQTT by measuring the accuracy of message clustering and comparing the inferred message formats with the ground truths on a traffic dataset captured from a campus. Our experimental results show that Inspector accurately cluster messages with 100% cluster precision and 100% message recall for TLS, 90% cluster precision and 99.6% message recall for SSH, 100% cluster precision and 92.7% message recall for MQTT. Based on the accurate message clusters, Inspector can correctly infer the format of the messages in the cluster.