Complex event processing for content-based text, image, and video retrieval

This report summarizes the findings of an exploratory team of the North Atlantic Treaty Organization (NATO) Information Systems Technology panel into Content-Based Analytics (CBA). The team carried out a technical review into the current status of theoretical and practical developments of methods, tools and techniques supporting joint exploitation of multimedia data sources. In particular, content-based information retrieval and analytics was considered as a means to allow military experts to exploit multiple data sources in a rapid fashion for sensemaking and knowledge generation. Elements included contextual understanding of complex events through computational/human processing techniques, event prediction through the automated extraction of network features, temporal trends, hidden clusters and resource flows, and the use of machine processing for automated translation, parsing, information extraction, and summarization of unstructured and semistructured data. The main conclusions of the study are that important research gaps exist in all the technical areas covered in this report. Though the research areas and developments are being advanced in the military sector and the civil sector, in particular, they remain at low levels of technical maturity for defense and security system applications. It is recommended that NATO collaborative research effort be expanded to advance those approaches that are most pertinent to our overall aim of enhancing the contextual understanding of complex events through CBA of heterogeneous multimedia streams.