The Astor-WISE system : A Federated Information Accumulator for Astronomy

The Astro-Wise consortium has designed a new paradigm, ‘Awe’, and implemented a fully scalable and distributed information system to overcome the huge information avalanche in wide-field astronomical imaging Surveys. In contrast to a TIER node setting, Awe allows the end-user to trace the data product, following all its dependencies up to the raw observational data and, if necessary, to re-derive the result with better calibration data and/or improved methods:‘Awe target processing’. This is achieved by: (i) emphasis on project management and documentation; (ii) translating the data model to an object model stored in a database; (iii) having all I/O reside in a distributed database, containing all metadata of bulk data, parameter values and catalogues; (iv) connecting to the database a federated file server that stores 100s hundreds of Terabytes of bulk data; (v) an own Awe compute-GRID which sends jobs to parallel clusters, which then request data from the database. The database architecture allows tools for rapid trend analysis, complex queries and fast hunting in Terabyte-sized catalogues. Thus, the system provides the user with transparent access to all stages of the data processing and thereby allows the data to be re-processed and add knowledge to the system, the Awe paradigm. All system components are distributed over Europe, enabling research groups to collaborate on shared projects. The web portal includes data viewing, quality labelling and compute-services. The Astro-Wise system will be connected to the Euro-VO, and publishing is effectively done by raising a flag by the Awe administrator. Currently, researchers use Awe with 10’s Tbyte of image data. Hundreds of Tbytes of data will start entering the system when the OmegaCAM camera starts operations in Chile. Several large surveys plan to use Awe: e.g., the 1500 Square Degree KIDS Survey (Kuijken et al.), the Vesuvio Survey of nearby superclusters, the OmegaWhite white dwarf binary survey, and OmegaTrans search for variables.