HIDStore: A Hierarchical Intermediate Data Storage System for Seismic Processing Application

Seismic processing is an important technology in petroleum industry. During the execution of seismic processing applications, large amount of intermediate data are generated and accessed. Providing high-performance services for intermediate data in the traditional storage architecture is expensive. In addition, because of the existence of new storage devices, the heterogeneity of storage environment has brought much inconvenience to the application developers and petroleum scientists. In this paper, we present a hierarchical intermediate data storage system called HIDStore. HIDStore employs distributed storage system based on the local storage devices and idle network resources to accelerate intermediate data access. Our experiments show that using HIDStore could improve the performance of various seismic processing applications and the resource utilization in compute cluster. HIDStore also abstracts different kinds of storage devices into hierarchical logical volumes and provides easy-to-use API to access data. Developers could deal with intermediate data in a high level of abstraction. Applications based on the HIDStore could fit into different storage environment and gain optimal performance automatically. Intermediate data in HIDStore could be automatically evicted once they expire.