Leaky Tiling Activations: A Simple Approach to Learning Sparse Representations Online.

Interference is a known problem when learning in online settings, such as continual learning or reinforcement learning. Interference occurs when updates, to improve performance for some inputs, degrades performance for others. Recent work has shown that sparse representations---where only a small percentage of units are active---can significantly reduce interference. Those works, however, relied on relatively complex regularization or meta-learning approaches, that have only been used offline in a pre-training phase. In our approach, we design an activation function that naturally produces sparse representations, and so is much more amenable to online training. The idea relies on the simple approach of binning, but overcomes the two key limitations of binning: zero gradients for the flat regions almost everywhere, and lost precision---reduced discrimination---due to coarse aggregation. We introduce a Leaky Tiling Activation (LTA) that provides non-negligible gradients and produces overlap between bins that improves discrimination. We empirically investigate both value-based and policy gradient reinforcement learning algorithms that use neural networks with LTAs, in classic discrete-action control environments and Mujoco continuous-action environments. We show that, with LTAs, learning is faster, with more stable policies, without needing target networks.