Representation Learning in Deep RL via Discrete Information Bottleneck

Several self-supervised representation learning methods have been proposed for reinforcement learning (RL) with rich observations. For real-world applications of RL, recovering underlying latent states is crucial, particularly when sensory inputs contain irrelevant and exogenous information. In this work, we study how information bottlenecks can be used to construct latent states efficiently in the presence of task-irrelevant information. We propose architectures that utilize variational and discrete information bottlenecks, coined as RepDIB, to learn structured factorized representations. Exploiting the expressiveness bought by factorized representations, we introduce a simple, yet effective, bottleneck that can be integrated with any existing self-supervised objective for RL. We demonstrate this across several online and offline RL benchmarks, along with a real robot arm task, where we find that compressed representations with RepDIB can lead to strong performance improvements, as the learned bottlenecks help predict only the relevant state while ignoring irrelevant information.

Download paper here

@article{DBLP:journals/corr/abs-2212-13835,
  author    = {Riashat Islam and
               Hongyu Zang and
               Manan Tomar and
               Aniket Didolkar and
               Md Mofijul Islam and
               Samin Yeasar Arnob and
               Tariq Iqbal and
               Xin Li and
               Anirudh Goyal and
               Nicolas Heess and
               Alex Lamb},
  title     = {Representation Learning in Deep {RL} via Discrete Information Bottleneck},
  journal   = {CoRR},
  volume    = {abs/2212.13835},
  year      = {2022},
}