Single-node partitioned-memory for huge graph analytics: cost and performance trade-offs

Because of cost, non-volatile memory NVDIMMs such as Intel Optane are attractive in single-node big-memory systems. We evaluate performance and cost trade-offs when using Optane as volatile memory for huge-graph analytics. We study two scalable graph applications with different work locality, access patterns, and parallelism. We evaluate single and partitioned address spaces---Memory and AppDirect modes---and compare with distributed executions on GPU-accelerated and CPU-based supercomputers. We show that AppDirect can perform and scale better than Memory for the largest working sets (12%), even when dominated by irregular access patterns, if most accesses are NUMA-local and Optane accesses are frequently reads. Surprisingly, between Memory and AppDirect, processor-cache performance can change due to line invalidations; updates to the caching policy (via non-temporal hints) can make a 25% improvement. We observe that single-node graph analytics frequently has >4--10× cost/performance advantages over distributed-memory executions on supercomputers.