A high‐performance distributed file system for large‐scale concurrent HD video streams

Summary With the rapid development of intelligent transportation technology, high-definition data storage, and processing of massive amounts of video surveillance have become key issues. When thousands of high-bit-rate video streaming concurrently writes, disk I/O throughput becomes a bottleneck. In addition, this leads to serious energy consumption and disk abrasion. To solve these problems, a new distributed file system for high concurrent and high-bit-rate writing is designed. It combines an optimized data storage model, efficient metadata management, and exquisite disk schedule mechanism. The optimized data storage model uses a file pre-allocation strategy and multiple-stream input modulating technology to convert the randomly concurrent writes on the disk into sequential writes; the metadata management provides an efficient means of retrieving the specified data; and the dual-partition schedule mechanism can ensure the disk stability with less abrasion. Through this distributed file system, the disk I/O throughput can be saturated in a high concurrent writing environment. The performance evaluation results demonstrate that the I/O throughput of a normal 7200RPM SATA III disk in our scheme can be stabilized at 150MB/s, easily to support 300 concurrent high-definition video streams (4Mbit/s each). The distributed file system with eight commodity servers can afford the ability of supporting 8000 high-definition video streams concurrently writing, which is far greater than the existing video surveillance storage solutions. Copyright © 2015 John Wiley & Sons, Ltd.