The existence of minimum pair of state and policy for Markov decision processes under the hypothesis of Doeblin

This paper studies the average-cost Markov decision process with compact state and action spaces and bounded lower semicontinuous cost functions. Following the idea of Borkar’s excellent papers [SIAMJ. Control Optim., 21 (1983), pp. 652–666; 22 (1984), pp. 965–978], the general case where irreducibility is not assumed is considered under the hypothesis of Doeblin and the existence of a minimum pair of state and policy, which attains the infimum of the average expected cost over all initial states and policies, is established. Further, it is proved that under additional weak conditions there exists an optimal stationary policy in the usual sense.