Big Data Platform for Oil and Gas Production Based on Apache Spark

For promoting the construction of intelligent oil field and applying big data technology to the daily production activities of oil field, this paper proposes a data mining platform based on Apache Spark multi-functional oil and gas production big data. This platform combines real-time and historical data processing-related framework and machine learning framework, designs massive oil and gas production data processing for different oil fields, supports real-time and offline processing data mining and other functions, supports the prediction of some data using related machine learning algorithms, supports the cleaning of large amounts of data, is used to train various deep learning models and will provide corresponding interfaces and permissions to relevant oil fields. Docker container is used to build the training environment of deep learning and machine learning, and the Kubernetes framework is used to complete the scheduling function. Struts, Spring and Hibernate relatively classic background processing framework is adopted at the Web framework level of the platform, different functions are decomposed into independent functional modules to reduce coupling, and agile development is used to improve the extended performance of the whole big data platform. This platform allows our data analysts to focus on data analysis and model training, without spending a lot of time and effort to deal with data problems, while the related calculation results are returned to other oil companies in an interface way or provide relevant operation authority to the field operation and maintenance personnel to access the data platform and data analysis.