NASA Earth observing system data and information system implementation of extra- and inter- file metadata : Application to data services

NASA's Earth Observing System (EOS) Data and Information System Core System (ECS) will become a very large repository for earth sciences data, which will include instrument measurements, derived model products, ancillary data from sources other than EOS, and so on. In addition to data archival functions, the system will also produce and distribute data. The needs of a heterogeneous community for services such as search, order, and geographic subsetting, has driven the selection of a common data format, Hierarchical Data Format (HDF), and a hierarchical metadata structure. The hierarchy consists of collection metadata which is a specification of attributes common to collections of data granules. A second tier of metadata, granule level, specifies attributes contained in individual data granules. A subset of these metadata reside in searchable databases, with the purpose of facilitating selection and retrieval of granules by both the production and the distribution sides of the system. All granule level metadata will be contained within their respective archived data files. Services that require manipulation of objects internal to individual granules are enabled through structural metadata. This third type of metadata provides information on the organization of granule objects as well as the relationship of objects to each other. EOS structural metadata contains information on components, such as name and scale factors; dimensions, specifying the size of arrays; and geolocation, describing the relationship of data to geolocation arrays. Structural metadata will be established by data production software invocation of an ECS-provided interface. This interface, called HDF-EOS, is an extension to the HDF interface. The metadata is formatted in Object Definition Language (ODL) and is written into the data files. Subsetting and other file manipulation services will be implemented using this metadata. In this paper, we will present an overview of the design, implementation, and operational aspects of metadata in the EOS. We will also discuss tools developed for assisting in the population of metadata in the operational system