FITS Tile Compression in the NOAO DMS
2009
The NOAO Data Management system (DMS) captures data from eleven NOAO and partner telescopes and transports these data from three mountaintops to replicate them between three data centers both North and South of the equator. Image files are annotated, remediated, ingested, and persisted through interfaces of the NOAO Science Archive. Wide-field optical and infrared images flow out of the archive, through the NOAO High Performance Pipeline creating several new data products that flow back into the archive. Raw, pipeline-reduced, and survey data products, both proprietary and postproprietary, are made available through the NOAO Portal using VO standards and services. Each of these several steps requires access to both image data and metadata in the form of image header keywords. Measures of storage efficiency and throughput characterize performance, cost, schedule, and risk in a matrix across all telescopes and all subsystems. Anything that impedes access to data or metadata diminishes throughput, thus slowing schedules, increasing costs, revealing risks, and adversely affecting performance. The familiar gzip compression algorithm is often used to increase data storage efficiency. However, gzip actually reduces throughput due to initial and recurring overhead of compression and later uncompression. For example, if metadata for an image require remediation, the whole image must be compressed, uncompressed, and compressed again. By contrast, the FITS tile convention using the Rice algorithm achieves about 40% better compression than gzip in just one-third the time. Image headers remain readable such that images often need never be uncompressed at all; metadata can be simply edited in place. Further, a library such as CFITSIO can support tile compression as a native image format. The pixel tiling feature means that for applications such as a cutout service, only the tiles overlapping the desired image section need be uncompressed. 1. Description of the Problem Figure 1 depicts a typical workflow between telescope, archive and user. A data acquisition system controls a camera leading up to the creation of the first digital representation of an observation. Observations occur in sequences, and the cadence of the resulting digital datastream varies throughout the observing session. The great majority of astronomical data are represented using the FITS format, usually as single or multiple image arrays. NOAO data flow between six sites on two continents. Similarly extensive routing applies to other major space and ground-based observatories. After the first copy is written in the dome, it must be transferred to an archival safe store. This data server is sited local to the mountaintop for reasons of network availability, and serves multiple telescopes for reasons of economy. Data sitting
Keywords:
- Correction
- Cite
- Save
- Machine Reading By IdeaReader
0
References
0
Citations
NaN
KQI