Stochastic simulation optimization benchmarking method in consideration of finite period of service.

Stochastic1 Simulation Optimization (SSO) is one of the most useful decision-making tools for those who simulate the real-world problems in which uncertainty appear, such as manufacturing or logistics issues. In recent years many SSO algorithms, such as stochastic surrogate model, optimal computing budget allocation, and various evolutionary optimization algorithms, have been developed. We have exploited test functions for benchmarking the algorithms, in which a deterministic part whose optimal values are known is accompanied by stochastic noise with zero expected value. Using the test functions, we have evaluated the algorithm in the viewpoint of closeness between the obtained fitness and the optimal value, which indicates how good the obtained solution will be for infinitely long period since the fitness converges to the deterministic value at infinity. In the real-world decision making, however, we often use the obtained solution during the finite service time, and after that, we make a decision again. From the perspective of real-world application, we propose a method to evaluate the SSO algorithms considering finite period of service. We illustrate that the optimal solution may differ depending on the period of service when the simulation has heterogeneous noise, which may influence the performance ranking of the algorithms. Using the well-known COCO benchmarking platform, we carry out the numerical experiments on noisy test functions, and show how the service time affects the performance.