Principled Offline RL in the Presence of Rich Exogenous Information

Learning to control an agent from offline data collected in a rich pixel-based visual observation space is vital for real-world applications of reinforcement learning (RL). A major challenge in this setting is the presence of input information that is hard to model and irrelevant to controlling the agent. This problem has been approached by the theoretical RL community through the lens of exogenous information, i.e., any control-irrelevant information contained in observations. For example, a robot navigating in busy streets needs to ignore irrelevant information, such as other people walking in the background, textures of objects, or birds in the sky. In this paper, we focus on the setting with visually detailed exogenous information and introduce new offline RL benchmarks that offer the ability to study this problem. We find that contemporary representation learning techniques can fail on datasets where the noise is a complex and time-dependent process, which is prevalent in practical applications. To address these, we propose to use multi-step inverse models to learn Agent-Centric Representations for Offline-RL (ACRO). Despite being simple and reward-free, we show theoretically and empirically that the representation created by this objective greatly outperforms baselines.

Download paper here

@inproceedings{DBLP:conf/icml/IslamTLEZDMLSC023,
  author       = {Riashat Islam and
                  Manan Tomar and
                  Alex Lamb and
                  Yonathan Efroni and
                  Hongyu Zang and
                  Aniket Rajiv Didolkar and
                  Dipendra Misra and
                  Xin Li and
                  Harm van Seijen and
                  Remi Tachet des Combes and
                  John Langford},
  editor       = {Andreas Krause and
                  Emma Brunskill and
                  Kyunghyun Cho and
                  Barbara Engelhardt and
                  Sivan Sabato and
                  Jonathan Scarlett},
  title        = {Principled Offline {RL} in the Presence of Rich Exogenous Information},
  booktitle    = {International Conference on Machine Learning, {ICML} 2023, 23-29 July
                  2023, Honolulu, Hawaii, {USA}},
  series       = {Proceedings of Machine Learning Research},
  volume       = {202},
  pages        = {14390--14421},
  publisher    = ,
  year         = {2023}
}