Design and Implementation of Air Quality Data Processing System Based on Big Data Technology

In this article, we design and implement an air quality data processing system based on big data technology, which leverages Flume to acquire data, Kafka to cache data, and Storm to process data, and finally save the results to the in-memory database called Redis and distributed database called HBase. In Redis, only the latest data of each monitoring point is saved, and the original data is deleted periodically to reduce the memory load. An application is implemented by which people can get real-time data from Redis and query historical data from HBase. Through experiments, it is proved that querying real-time data from Redis is faster than relational database. The system solves the issues of poor storage capacity and slow query speed when using relational database in the big data environment, and improves the efficiency of data processing significantly.