Partition-Aware Scalable Outlier Detection Using Unsupervised Learning

Anomaly Detection is an important area of research in data science. However, for big data it is challenging due to limited (or no) labeled data during training. This challenge is prevalent in the cyber-security area. There is always a lag before labels of (near) real time data become available. Hence, we would like to develop a new anomaly detection framework without any labeled data (no human intervention). For this, we utilize the density-based spatial clustering of applications with noise (DBSCAN). To address the scalability issues that arise when using DBSCAN, we exploit novel techniques based on partitions with minimum false alarm. We compare our approaches with other unsupervised approaches to demonstrate the effectiveness of our work.