Handling Large Numerical Data-Sets: Viability of a Lossy Compressor for CFD-simulations

Over the years, a steady increase in computing power has enabled scientists and engineers to develop increasingly complex applications for machine learning and scientific computing. But while these applications promise to solve some of the most difficult problems we face today, their data hunger also reveals an ever-increasing I/O bottleneck. It is therefore imperative that we develop I/O strategies to better utilize the raw power of our high-performance machines and improve the usability and efficiency of our tools. To achieve this goal, we have developed the BigWhoop compression library based on the JPEG 2000 standard. It enables the efficient and lossy compression of numerical data-sets while minimizing information loss and the introduction of compression artifacts. This paper presents a comparative study using the Taylor-Green Vortex test case to demonstrate the superior compression performance of BigWhoop compared to contemporary solutions. An evaluation of compression-related distortion at high compression ratios is shown to prove its feasibility for both visualization and statistical analysis.