A Dynamic Holding Approach to Stabilizing a Bus Line Based on the Q-learning Algorithm with Multistage Look-ahead.

The unreliable service and the unstable operation of a high frequency bus line are shown as bus bunching and the uneven distribution of headways along the bus line. Although many control strategies, such as the static and dynamic holding strategies, have been proposed to solve the above problems, many of them take on some oversimplified assumptions about the real bus line operation. So it is hard for them to continuously adapt to the evolving complex system. In view of this dynamic setting, we present an adaptive holding method which combines the classic approximate dynamic programming (ADP) with the multi-stage look-ahead mechanism. The holding time, that is the only control means used in this study, will be determined by estimating its impact on the operation stability of the bus line system in the remained observation period. The multi-stage look-ahead mechanism introduced into the classic Q-learning algorithm of the ADP model makes the algorithm get through its earlier unstable phase more quickly and easily. During the implementation of the new holding approach, the past experiences of holding operations can be cumulated effectively into an artificial neural network used to approximate the unavailable Q-factor. The use of a detailed simulation system in the new approach makes it possible to take into accounts most of the possible causes of instability. The numerical experiments show that the new holding approach can stabilize the system by producing evenly distributed headway and removing bus bunching thoroughly. Comparing with the terminal station holding strategies, the new method brings a more reliable bus line with shorter waiting times for passengers.