Improved MPI Multi-Threaded Performance using OFI Scalable Endpoints

Message Passing Interface (MPI) applications are launched as a set of parallel homogeneous processes, commonly with one to one mapping between MPI processes and compute cores. With the growing complexity of MPI applications and compute node processors consisting of large numbers of cores, launching a small number of MPI processes with several lightweight threads per process is becoming popular. Task based programming models in combination with MPI also provide several benefits for the application to exploit intra node parallelism. Naive implementation of MPI THREAD MULTIPLE can be expensive with minimal or no performance benefits. We demonstrate a high-performance end to end multi-threading solution across the MPI application and MPI runtime, with threads mapping to hardware resources. We demonstrate our solution with Open MPI using Libfabric (a.k.a. OpenFabrics Interfaces OFI) and its Intel R Omni-Path Performance Scaled Messaging 2 (PSM2) provider. Our tests with Intel MPI Benchmarks Multi Thread set (IMB-MT) show BW improvement for large message sizes when running with multiple threads. We also demonstrate up to 2.5x performance improvements with Baidu All-Reduce. Even though the experiments were run on Intel R Omni-Path Architecture fabric, the solution can be applied to other fabrics with the capability of allocating resources among multiple threads.