Hierarchical reinforcement learning for self‐driving decision‐making without reliance on labelled driving data

TLDR

Self‑driving car decision making is typically addressed by rule‑based systems or supervised imitation learning, both of which require extensive labelled driving data. This study proposes a hierarchical reinforcement learning approach that eliminates the need for large amounts of labelled driving data. The approach decomposes driving into three manoeuvres—lane keeping, right lane change, and left lane change—learning a sub‑policy for each and a master policy that selects the appropriate manoeuvre, with all policies implemented as fully‑connected neural networks trained via asynchronous parallel reinforcement learning using distinct state spaces and reward functions. When applied to a highway driving scenario, the method achieved smooth and safe decision making.

Abstract

Decision making for self‐driving cars is usually tackled by manually encoding rules from drivers’ behaviours or imitating drivers’ manipulation using supervised learning techniques. Both of them rely on mass driving data to cover all possible driving scenarios. This study presents a hierarchical reinforcement learning method for decision making of self‐driving cars, which does not depend on a large amount of labelled driving data. This method comprehensively considers both high‐level manoeuvre selection and low‐level motion control in both lateral and longitudinal directions. The authors firstly decompose the driving tasks into three manoeuvres, including driving in lane, right lane change and left lane change, and learn the sub‐policy for each manoeuvre. Then, a master policy is learned to choose the manoeuvre policy to be executed in the current state. All policies, including master policy and manoeuvre policies, are represented by fully‐connected neural networks and trained by using asynchronous parallel reinforcement learners, which builds a mapping from the sensory outputs to driving decisions. Different state spaces and reward functions are designed for each manoeuvre. They apply this method to a highway driving scenario, which demonstrates that it can realise smooth and safe decision making for self‐driving cars.