OpenMP versus MPI for PDE solvers based on regular sparse numerical operators

Two parallel programming models represented by OpenMP and MPI are compared for PDE solvers based on regular sparse numerical operators. As a typical representative of such an operator, a finite difference approximation of the Euler equations for fluid flow is considered. The comparison of programming models is made with regard to uniform memory access (UMA), non-uniform memory access (NUMA), and self-optimizing NUMA (NUMA-opt) computer architectures. By NUMA-opt, we mean NUMA systems extended with self-optimization algorithms, in order to reduce the non-uniformity of the memory access time. The main conclusions of the study are: (1) that OpenMP is a viable alternative to MPI on UMA and NUMA-opt architectures; (2) that OpenMP is not competitive on NUMA platforms, unless special care is taken to get an initial data placement that matches the algorithm; (3) that for OpenMP to be competitive in the NUMA-opt case, it is not necessary to extend the OpenMP model with additional data distribution directives, nor to include user-level access to the page migration library.