Test Your Self-Driving Algorithm: An Overview of Publicly Available Driving Datasets and Virtual Testing Environments

TLDR

Autonomous driving systems aim for SAE Level 5, but their complex software demands a comprehensive testing strategy. This paper extends a 2017 survey of publicly available driving datasets by adding new datasets and summarizing virtual testing environments for closed‑loop evaluation. Open‑loop prototyping uses recorded sensor data, while closed‑loop testing requires real‑vehicle trials or virtual environments, which the authors catalog and integrate into their framework. The survey identified 37 open‑loop datasets and 22 virtual testing environments, demonstrating that community resources substantially support perception research and end‑to‑end simulation.

Abstract

Many companies aim for delivering systems for autonomous driving reaching out for SAE Level 5. As these systems run much more complex software than typical premium cars of today, a thorough testing strategy is needed. Early prototyping of such systems can be supported using recorded data from onboard and surrounding sensors as long as open-loop testing is applicable; later, though, closed-loop testing is necessary-either by testing on the real vehicle or by using a virtual testing environment. This paper is a substantial extension of our work presented at the 2017 IEEE International Conference on Intelligent Transportation Systems (ITSC) that was surveying the area of publicly available driving datasets. Our previous results are extended by additional datasets and complemented with a summary of publicly available virtual testing environments to support closed-loop testing. As such, a steadily growing number of 37 datasets for open-loop testing and 22 virtual testing environments for closed-loop testing have been surveyed in detailed. Thus, conducting research toward autonomous driving is significantly supported from complementary community efforts: A growing number of publicly accessible datasets allow for experiments with perception approaches or training and testing machine-learning-based algorithms, while virtual testing environments enable end-to-end simulations.