Real-Time digital libraries based on widely distributed, high performance management of large data objects

We describe a distributed, wide area network based approach to collecting, cataloguing, storing, and providing Web access for large-data-objects that origi- nate as high-speed data streams. Such data streams result from the operation of many types of online instruments and imaging systems, and are a ''staple'' of modern intelligence, scientific, and health care environments. The approach provides for real-time conversion of the data streams and large datasets to ''objects'' that are manage- able, extensible, searchable, browsable, persistent, and available to both ''ordinary'' and high-performance applications through the integration of a high-speed distributed cache and transparent management of tertia- ry storage. The user interfaces - for both application users and data collection curators - are provided by the capabilities of the World Wide Web. The capabilities of the architecture are not unlike a digital library system, and we give an example of a digital image library that has been built using this architecture. However, our ap- proach particularly addresses the issues involved in creating such digital library-like collections automatically from the high date-rate, real-time output of, e.g., satellite imaging systems, scientific instruments, health care imaging systems, etc. We discuss the capabilities, archi- tecture, and implementation of such a system, as well as several example applications in data-intensive environ- ments. The applications include a metropolitan area ATM network based, online health care video imaging system, and several image database applications, includ- ing a photographic-image library (see (9)). We also describe the security architecture used to enforce data owner imposed access conditions.