Research on Data Flow Partitioning Based on Dynamic Feature Extraction

With the rapid development of the Internet of things, social networks, and e-commerce, the era of big data has arrived. Although big data has great potential for many areas such as industry, education, and health care, getting valuable knowledge from big data can be a daunting task. Big data has the characteristics of high-speed change, and its content and distribution characteristics are in dynamic changes. Most current models are static learning models that do not support online updating, making it difficult to learn dynamically changing big data features in real time. In order to solve this problem, this paper proposed a method to support incremental recursive least squares (IRLS) regression parameter estimation and variable sliding window algorithm to analyze and judge the trends of dynamic characteristics of data streams, which can provide early warning, status assessment, and decision support for monitoring objects and improve the accuracy and adaptability of data flow classification. The real-time computational and analysis accuracy are obviously improved than the traditional algorithm, and the simulation results verify the effectiveness of the proposed algorithm.