OpenSHMEM Sets and Groups: An Approach to Worksharing and Memory Management

Collective operations in the OpenSHMEM programming model are defined over an active set, which is a grouping of (PEs) based on a triple of information including the starting PE, a \(log_2\) stride, and the size of the active set. In addition to the active set, collectives require Users to allocate and initialize synchronization (i.e., pSync) and scratchpad (i.e., pWrk) buffers for use by the collective operations. While active sets and the user-defined buffers were previously useful based on hardware and algorithmic considerations, future systems and applications require us to re-evaluate these concepts. In this paper, we propose Sets and Groups as abstractions to create persistent, flexible groupings of PEs (i.e., Sets) and couple these groups of PEs with memory spaces (i.e., Groups), which remove the allocation and initialization burden from the User. To evaluate Sets and Groups, we perform multiple micro-benchmarks to determine the overhead of these abstractions and demonstrate their utility by implementing a distributed APSP application, which we evaluate using multiple synthetic and real-world graphs.