UStore: An optimized storage system for enterprise data warehouses at UnionPay

UnionPay's inter-bank transaction settlement platform (ITSP) generates a huge amount of bankcard transaction data everyday, recording different bankcard activities. In order to unleash the business value of these data, UnionPay has built a customized data warehouse based on Hadoop to manage and query the massive data imported from ITSP. However, the original system suffers from low storage utilization due to various types of data redundancy. Such data redundancy is caused by the long-term evolution of the system architecture. It dramatically wastes storage space, degrades query performance and leads to data inconsistency problem. In order to address these issues, we have developed UStore, an optimized storage system to reduce most data redundancies and improve query performance. In this paper, we present the design and implementation of UStore in detail. We test the performance of UStore on UnionPay's real data and the results show significant improvements in both storage utilization and query performance. To date, UStore has been deployed to process over 15 years' bankcard transaction data (over 3PB in plain text format) in UnionPay.