Real-time time series error-based data reduction for internet-of-things applications

There are many time series data reduction methods, ranging from primitive data aggregation such as Rate of Change to sophisticated compression algorithms. Unfortunately, many of these existing algorithms are limited to work in offline mode only, data can only be reduced after a certain amount of data is collected. Such offline mode is not suitable for IoT applications such as monitoring, surveillance and alert system which needs to detect events at real-time. On the other hand, existing real-time time series data reduction techniques often require manual configuration and adaption to intended applications and hardware like IoT gateway. Such requirements prevent effective deployments of data reduction techniques. This work is inspired by Perceptually Important Points (PIP) data reduction algorithm due to its superior data reduction ability. This work differs from existing PIP in the sense that, we have devised a real-time data reduction algorithm namely error-based PIP Data Reduction (PIPE), that operates with a single value configuration; error rate, which can be used with various sensor data without any priori analysis required. In additional to that, PIPE is simple to the extent that it can be deployed at the sensor node as well. Through 7 different time series datasets and by comparing the result against the existing data reduction techniques such as GZIP, Real-Time PIP and Rate of Acceleration threshold-based data reduction, the experimental results are promising, the evaluation shows that it is possible that by only forwarding 10% of data, the reduced data produced by PIPE can be used to reconstruct the time series with an accuracy of 0.98 in real-time.