A multi-level approach for understanding I/O activity in HPC applications

I/O has become one of the determining factors of HPC application performance. Understanding an application's I/O activity requires a multi-level view of the I/O function flow that includes high-level I/O libraries. We have developed a tracing framework, called Recorder, that captures I/O function calls at multiple layers of the parallel I/O stack without requiring source code modifications. In this paper, we show how Recorder's trace output can be used to investigate I/O activity and identify performance inefficiencies in two I/O benchmarks running on a leading edge HPC platform. Future work to organize and present the collected information more intuitively will further increase the value of Recorder's capabilities. We believe that a multi-level I/O tracing framework can provide key insights to end users and I/O library developers working to improve I/O on HPC platforms.