A Hardware Accelerator for Language Guided Reinforcement Learning

Reinforcement learning (RL) has shown great performance in solving sequential decision-making problems. This paper proposes a framework to train RL agents conditioned on constraints that are in the form of structured language to improve training efficiency. We implemented an energy-efficient hardware accelerator to receive both images and text inputs that allows RL agents understand human language and act in real-world environments. A scalable and parallel hardware with different number of processing elements is implemented on both FPGA and ASIC that provides a balance between power consumption and performance. The post-layout ASIC design consumes 9.2 mW while providing 361 fps throughput.