Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

TLDR

The paper presents a pipeline that parametrically samples and renders static multi‑task vision datasets from comprehensive real‑world 3D scans. The pipeline is Dockerized, offers a CLI, Python code, PyTorch dataloaders, a starter dataset, download scripts, and utilities, all available via the project website. Training on the generated starter dataset yields state‑of‑the‑art performance on multiple vision tasks, with depth estimation surpassing MiDaS and surface normal estimation achieving human‑level results on the OASIS benchmark.

Abstract

This paper introduces a pipeline to parametrically sample and render static multi-task vision datasets from comprehensive 3D scans from the real-world. In addition to enabling interesting lines of research, we show the tooling and generated data suffice to train robust vision models. Familiar architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks, despite having seen no benchmark or non-pipeline data. The depth estimation network outperforms MiDaS and the surface normal estimation network is the first to achieve human-level performance for in-the-wild surface normal estimation—at least according to one metric on the OASIS benchmark. The Dockerized pipeline with CLI, the (mostly python) code, PyTorch dataloaders for the generated data, the generated starter dataset, download scripts and other utilities are all available ${\color{Magenta}through}\;{\color{Magenta}our}\;{\color{Magenta}project}\;{\color{Magenta}website}$.