An Advanced Driving Agent with the Multimodal Large Language Model for Autonomous Vehicles

Abstract

As deep learning technology advances Autonomous Driving (AD), existing AD methods encounter performance limitations, especially in handling corner cases, interpretability, and verifiability, which are crucial for the safety of connected and autonomous vehicles. Multimodal Large Language Models (MLLMs) demonstrate remarkable understanding and reasoning capabilities, presenting a transformative opportunity to overcome challenges faced by traditional AD algorithms. We conduct a comprehensive study on the application of MLLMs in AD, exploring their potential to address critical challenges faced by traditional AD algorithms. We construct a Visual-Question-Answering dataset for model fine-tuning to address hallucinations and poor logic analysis issues in MLLMs. We then decompose the AD decision-making process into Scene Understanding, Prediction, and Decision, allowing MLLMs to construct Chain-of-Thought to make decisions step by step. Subsequently, we propose a new framework enabling models to perform AD tasks under conditions of limited local computing resources, few-shots, multimodality, and complex scenarios, enhancing the flexibility of future AD system deployment. Our extensive experiments and in-depth analyses demonstrate the significant advantages of MLLMs for AD. We also discuss the strengths and weaknesses of existing methods, providing a detailed outlook on MLLMs in AD.