Symmetric Memory Partitions in OpenSHMEM: A Case Study with Intel KNL

To extract best performance from emerging tiered memory systems, it is essential for applications to use the different kinds of memory available on the system. OpenSHMEM memory model consists of data objects that are private to each Processing Element (PE) and data objects that are remotely accessible by all PEs. The remotely accessible data objects are called Symmetric Data Objects and are allocated on a memory region called as Symmetric Heap. Symmetric Heap is created during program execution on a memory region determined by the OpenSHMEM implementation. This paper proposes a new feature called Symmetric Memory Partitions to enable users to determine the size along with other memory traits for creating the symmetric heap. Moreover, this paper uses Intel KNL processors as an example use case for emerging tiered memory systems. This paper also describes the implementation of symmetric memory partitions in Cray SHMEM and use ParRes OpenSHMEM microbenchmark kernels to show the benefits of selecting the memory region for the symmetric heap.