Visualizing the World Model Learned by an Agent’s Value Function

Summary

This study asks whether model-free reinforcement learning can still learn an internal model of its environment. The researchers use counterfactual interventions that change one targeted aspect of a state while keeping everything else fixed, then examine the value function of a Proximal Policy Optimization agent trained in the Crafter environment. Although the agent was trained only to optimize reward, its value function appears to represent several environmental concepts. These include the rules determining when a diamond can be collected, the effects of water and food on health, and heuristics related to inventory. The authors use visualizations of these representations to discuss potential safety implications. They also identify shortcomings of current reinforcement learning agents that may limit progress toward more advanced machine intelligence.