Anomaly and Degradation Detection Using Subspace Tracking in Streaming Data

In our increasingly connected digital world, many sensors are connected to each other. Each sensor provides several features and data from those features. As a result, the data sets created from Internet of Things (IoT) applications can consist of hundreds or even thousands of dimensions. Even though the dimensionality is high in these data sets, the actual rank is mostly low because of the high correlations between these dimensions. As a result, subspace analysis and subspace tracking are useful methods of capturing low-dimensional structures from high-frequency, high-dimensional data. In streaming data, a substantial change in the subspace can indicate an anomaly. In this paper, we introduce several measures for detecting changes in the subspace that we use in our anomaly and degradation detection methods. We also present the results of applying our methods to simulated and real-world data sets. These methods have been implemented in an event stream processing platform.