Information Selection in Intelligence Processing

Abstract : In many intelligence agencies, the processing of data into usable information ready for analysis poses a significant bottleneck. Typically, much more data is available than can be processed in the limited time available for processing. We formulate the problem faced by an intelligence collection unit when processing incoming raw information for delivery to intelligence analyst as an exploration-exploitation problem: the processor has to choose between exploring for new sources of relevant information and exploiting known sources. To address the exploration-exploitation problem, we develop a mathematical model of the processor's knowledge and examine several algorithms that allow the processor to maximize the discovery of relevant data given a time limit. The algorithms studied include the Pure Exploitation algorithm, the Softmax algorithm, the Value Difference Based Exploration algorithm, the Knowledge-Gradient Exploration First algorithm, and the Wide Exploration First algorithm. We derive insights on the performance of these algorithms using a simulated case study.