Data decomposition in Monte Carlo neutron transport simulations using global view arrays

Accommodating large tally data can be a challenging problem for Monte Carlo neutron transport simulations. Current approaches include either simple data replication, or are based on application-controlled decomposition such as domain partitioning or client/server models, which are limited by either memory cost or performance loss. We propose and analyze an alternative solution based on global view arrays. By using global view arrays, tallies are naturally partitioned into small globally addressable blocks that fit in the limited on-node memory of compute nodes, achieving both highly scalable memory and performance efficiency. This approach also greatly simplifies the programmability compared with application-controlled approaches. Our implementation is based on integrating a global view library built on MPI one-sided communication, global view resilience GVR, into the OpenMC Monte Carlo transport code. The remote memory access RMA-based global view array implementation is able to achieve 85% efficiency at 16,384 processes compared with 1,000 processes with 2.39 TB mesh tally across 1,366 nodes on a Cray XC30 supercomputer. Our results improve scalability significantly compared with the tally server approach and are better than any other published results, indicating that global view array is a promising alternative to enable full-core light water reactor analysis on current and future computer systems.