Real-Time Information Extraction from Big Data

Abstract : We are drowning under the 3 Vs (volume, velocity and variety) of big data. Real-time information extraction from big data, at global as well as local levels, is critical for making rapid decisions in many important DoD scenarios. However, such realtime information from big data is enormously complex and extremely challenging. We argue that data movement is of crucial importance in big data analytics, and show that recently developed photonic communication technologies has the potential to extract real-time information, and improve performance/Watt by a hundredfold with corresponding cost savings at the local, rack-size level, provided the entire system, consisting of communications hardware and network, computer hardware, algorithms, software and architecture, is holistically optimized. We provide examples of some applications of interest to DoD for which such holistic system optimization could be a game changer.