Efficient Learning in Stationary and Non-stationary OSA Scenario with QoS Guaranty

In this work, the opportunistic spectrum access (OSA) problem is addressed with stationary and non-stationary Markov multi-armed bandit (MAB) frameworks. We propose a novel index based algorithm named QoS-UCB that balances exploration in terms of occupancy and quality, e.g. signal to noise ratio (SNR) for transmission, for stationary environments. Furthermore, we propose another learning policy, named discounted QoS-UCB (DQoS-UCB), for the non-stationary case. Our contribution in terms of numerical analysis is twofold: i) In stationary OSA scenario, we numerically compare our QoS-UCB policy with an existing UCB1 and also show that QoS-UCB outperforms UCB1 in terms of regret and ii) in non-stationary OSA scenario, numerical results state that proposed DQoS-UCB policy outperforms other simple UCBs and also QoS-UCB policy. To the best of our knowledge, this is the first learning algorithm which adapts to non-stationary Markov MAB framework and also quantifies channel quality information.