Online Data Extraction for Large-Scale Agent-Based Simulations

Cloud-based simulation systems reduce the upfront hardware costs of running high-performance experiments and increases the ease with which simulation experiments can be repeated. The data being generated by simulations can be large. Commonly used data storage systems such as relational databases can handle large amounts of data, but the analysis is a challenging problem. Moreover, handling this amount of data in cloud services can be both expensive (bandwidth and storage costs) and time-consuming. However, a lot of the data that is generated by agent-based simulations does not contribute directly to the purpose of the experiment being conducted. We propose an extension to cloud-based simulation systems that rather than storing raw simulation output data, uses stream data processing to generate the result dataset while the simulation is running. This can then be used to store only the data required for later use, this saving both time and money.