Characterizing Error in the Verification Procedure of the ICESat-2 ATLAS Instrument’s Level-1B Product

The Advanced Topographic Laser Altimeter System (ATLAS) is the sole science instrument on NASA's ICESat-2 mission. It was designed to measure elevation simultaneously along six tracks on the earth's surface with centimeter-level vertical precision, demanding a picosecond-level precision in photon time of flight. To ensure this precision requirement was met, we developed for the Level-1B ATLAS data product a careful verification procedure. To quantify the amount of acceptable error we needed to understand the effects of the various floating-point precisions at all steps of the calculations and the limitations of the programming languages used for the production software (Fortran) and for the verification software (Python). For example, we found in the 64-bit photon time-of-flight calculations that differences even at 13 or 14 decimal places often revealed that an incorrect calibration value had been selected. Without first characterizing the acceptable error, such small errors could be overlooked and could...