Multi-agent Reinforcement Learning for Green Energy Powered IoT Networks with Random Access

Energy harvesting is a promising solution to enable energy sustainable operation of IoT devices. Especially for under-water IoT network as it is difficult and costly for underwater IoT devices to replace the battery. Unlike traditional power supply, energy harvesting from green sources is a random process and is dependent on the charging environment, which poses new challenges for provisioning quality of services of IoT networks. Due to the high cost for low-powered IoT devices to update its energy status with the scheduler, distributed transmission protocol is more desirable for the IoT networks. In this work, we consider an IoT network where IoT devices use adaptive p-persistent ALOHA for data transmissions. Each IoT device can contend for channel access only when it is ready, i.e., it has a data for transmission and it harvests enough energy for communications. Due to stochastic energy harvesting and random access, the number of ready devices in the network may vary. As such, an analytical framework is first developed using a discrete Markov model to analyze the average number of ready devices. Next, an optimization problem is formulated to maximize the system throughput by tuning the transmission probability. Given that the wireless environment is unknown at different IoT devices, e.g., total number of contending devices, data arrival rates of other IoT devices, a multi-agent reinforcement learning algorithm is introduced for each device to autonomously tune the transmission probability in a distributed manner. In addition, game theory is applied to design the reward function to ensure an equilibrium and to closely approach the optimal parameter setting. Numerical results show that the proposed learning algorithm can greatly improve the throughput performance comparing with other algorithms.