Knowledge Transfer from Keepaway Soccer to Half-field Offense through Program Symbiosis

Abstract

Half-field Offense (HFO) is a sub-task of Robocup 2D Simulated Soccer. HFO is a challenging, multi-agent machine learning problem in which a team of offense players attempt to manoeuvre the ball past a defending team and around the goalie in order to score. The agent's sensors and actuators are noisy, making the problem highly stochastic and partially observable. These same real-world characteristics have made Keepaway soccer, which represents one sub-task of HFO, a popular testbed in the reinforcement learning and task-transfer literature in particular. We demonstrate how policies initially evolved for Keepaway can be reused within a symbiotic framework for coevolving policies in genetic programming (GP), with no additional training or transfer function, in order to improve learning in the HFO task. Moreover, the highly modular policies discovered by GP are shown to be significantly less complex than solutions based on traditional value-function optimization while achieving the same level of play in HFO.