A Stream Processor for Extracting Usage Intelligence From High-Momentum Internet Data

The data streams of the Internet are quite large and present significant challenges to those wishing to analyze these streams on a continuous basis. Opportunities for analysis for a Network Service Provider include understanding subscriber usage patterns for developing new services, network demand flows for network operations and capacity planning functions, and early detection of network security breaches. The conventional analysis paradigm of store first, then analyze later has significant cost and latency issues when analyzing these high-momentum streams. This article presents a deployed architecture for a general purpose stream processor that includes dynamically configurable Capture Models that can be tailored for compact collection of statistics of the stream in real time. The highly configurable flow processing model is presented with numerous examples of how multiple streams can be merged and split based on the requirements at hand.