Reinforcement Learning for RoboCup Soccer Keepaway

Peter Stone, Richard S. Sutton, Gregory Kuhlmann

Adaptive Behavior · 2005 · 390 citations · 39 references

Concepts

TL;DR

RoboCup simulated soccer challenges reinforcement learning with a large state space, hidden states, concurrent multi‑agent learning, and variable action delays. The study applies episodic SMDP Sarsa(λ) with linear tile‑coding and variable λ to learn higher‑level decisions in RoboCup keepaway. Using this approach, agents learn when to hold or pass the ball, and the method is tested across varying field sizes and team numbers. The resulting policies significantly outperform a range of benchmark strategies.

Abstract

RoboCup simulated soccer presents many challenges to reinforcement learning methods, including a large state space, hidden and uncertain state, multiple independent agents learning simultaneously, and long and variable delays in the effects of actions. We describe our application of episodic SMDP Sarsa(λ) with linear tile-coding function approximation and variable λ to learning higher-level decisions in a keepaway subtask of RoboCup soccer. In keepaway, one team, “the keepers,” tries to keep control of the ball for as long as possible despite the efforts of “the takers.” The keepers learn individually when to hold the ball and when to pass to a teammate. Our agents learned policies that significantly outperform a range of benchmark policies. We demonstrate the generality of our approach by applying it to a number of task variations including different field sizes and different numbers of players on each team.

References

39