Challenges in Flexible Aggregation of Pervasive Data

The vision of billions of users connected to millions of services using trillions of devices is fast becoming a reality. The result will be a vast network of mobile communication devices and data sources, including sensors, newsfeeds, web services, and databases. Potential uses of this data span a wide range of application domains, including medical monitoring, traffic routing, proximity detection, electricity management, and service-fleet dispatching. Applications require flexible mechanisms for constructing condensed and refined views of the raw data, possibly in ways unanticipated by the data providers. Aggregation comprises collection of high volumes of raw data from data sources, composition of the raw data into less voluminous refined data, and timely delivery of the refined data to applications. There are difficult challenges inherent in creating an aggregation system that is sufficiently flexible, scalable, and reliable to address the needs of applications. 1 The aggregation problem The explosive growth of networked data sources, including sensors, cellular telephones, online services, databases, and data feeds, presents novel opportunities for timely use of the data. The data is heterogeneous and possibly voluminous. Applications will combine raw data from multiple data sources in diverse ways, in a flexible, scalable, and reliable manner. Aggregation comprises the collection of raw data from pervasive data sources, the flexible, programmable composition of the raw data into less voluminous refined data, and the timely delivery of the refined data to data consumers. We propose the construction of an aggregation system to manage shared resources and facilitate the writing of applications. Aggregation of data from many known sources in a predetermined manner, although difficult, is not the foremost challenge we face. The need for flexibility makes the problem especially challenging. Flexibility has three key aspects: ŸPervasive data sources cannot always anticipate possible uses of the data, and applications cannot always predetermine the sources of data that can be used for a computation. ŸComposition of data must be flexible to accommodate diverse characteristics of data sources, mobility of data sources and consumers, and the dynamic nature of network resources. ŸScalability and reliability are intertwined with flexibility: An aggregation system must flexibly choose data sources and manners of composition in order to be both scalable and reliable. As an example, consider an application that tells drivers the best routes from their current positions to their destinations. The computation is based on the driver’s current location and