Issues in Data Management in Observing Systems and Lessons Learned

With ocean observatories growing in importance and several development efforts underway, it is critical to understand the goals and issues that an ocean observing system will have to face and solve. There are many components such as network infrastructure, instrumentation, control systems, data systems and client applications that will need to interoperate seamlessly in order for the observatory to effectively meet the needs of the research community, policy makers, and the general public. For example, instruments serve as the critical foundation to the usefulness of the information extracted from an ocean observatory. However, configuring these instruments for deployment in an observatory can be a time consuming and error-prone task. Although manageable on an instrument-by-instrument basis, configuration becomes a very important issue as the size of the observatory and the number of instruments it operates grows. Each instrument type also has its unique power, communication and bandwidth requirements that further complicate their integration into observatory systems. As the complexity of integrating the instruments into the observatory increases, so does the overall operating cost of the observatory, thus affecting the overall capability of the system. For this reason, adding and removing instruments needs to be as simple as possible, which necessitates that the infrastructure handle a large portion of that integration automatically. Once these instruments are successfully deployed, the infrastructure must also be able to monitor the health and status of the various observatory assets. Once past the configuration and management issues of instruments, the observatory still faces other issues from this collection of data from these heterogeneous instruments. Metadata and data management are particularly difficult problems to handle from a systematic perspective. Being able to capture and utilize metadata and data is certainly one requirement of a data management system, but having it operate in an automated and robust way creates even more complications. The metadata associated with data must be correctly captured by the system but also maintained correctly and linked with other related metadata throughout the system. Through this metadata, the relationship between the data and its context (source, environment, location, etc) can be captured and utilized in the analysis of the data in search of various phenomenon like events, trends, patterns, etc. Adding to these complicated tasks is the requirement that this data system must interoperate with other systems as both a server and a client. Even with these high level goals and issues defined, there is no substitute for practical experience in affirming that the right goals and issues were identified. At the Monterey Bay Aquarium Research Institute (MBARI), we have had an active ocean observing development project (Monterey Ocean Observing System-MOOS) going for the past several years. With that experience, we have been able to identify several key issues and lessons learned that are relevant to ocean observatories both from the development and operational perspectives. This paper will describe the different goals that the MOOS system and its associated data management system, the Shore-Side Data System (SSDS) address. Specifically, we will discuss how the SSDS handles various data and metadata related issues and what is gained by solving those issues. Practical examples of these solutions will be given and they will be used to illustrate how, and why, certain issues are important for the data management system to address. As a final wrap up, a section on lessons learned will be discussed to help transition what we have learned to the general oceanographic community