The Discounted Cost Model

In this chapter, let \(\alpha >0\) be a fixed discount factor. We shall consider the \(\alpha \)-discounted CTMDP problems ( 1.15) and ( 1.16). Without loss of generality we can restrict to (relaxed) \(\pi \)-strategies because (Markov) \(\pi \)-strategies are sufficient for \(\alpha \)-discounted problems; see Remark 1.3.1. (In fact, in this chapter, the class of natural Markov strategies is also sufficient, according to ( 2.92). However, we choose to stay with the more general class of \(\pi \)-strategies. The same is true for some of the subsequent chapters.) Although most \(\pi \)-strategies are not realizable, under appropriate conditions, the solution to the unconstrained problem is given by a realizable deterministic stationary strategy (see Theorem 3.1.2). Moreover, as was explained in Sect. 1.3.5, for every \(\pi \)-strategy there exists an equivalent standard \(\xi \)-strategy which is realizable.