Research and Optimization of Massive Music Data Access Based on HDFS

To build a storage management platform for music big data, we need to collect massive heterogeneous music resources from the Internet and store them on big data platforms. Therefore, it is a key problem to build a storage management system that is high performance, extensible, scalable and capable of supporting big data. Building big data platform based on HDFS is a feasible scheme. However, HDFS has good performance for accessing large files, but it is very inefficient for small files such as music source files and music metadata. In view of this, this paper proposes a geared to the needs of mass music based on HDFS small file access optimization scheme, using the format of the music data classification, build multistage merger queue, merging small files into large files in order to reduce the number of files, and create an indexing mechanism to access small files. The index file is then stored in the Phoenix + HBase storage repository and associated with the music metadata to improve the reading efficiency of music small files and music metadata. The experimental test verifies the effectiveness of the optimized scheme and meets the demand of music big data storage management platform.