Single image depth estimation from predicted semantic labels

TLDR

Depth cues are weaker at distance, as exemplified by the more uniform appearance of trees far away. The study aims to estimate per‑pixel depth from a single monocular image. The authors first perform semantic segmentation and then use the resulting labels to guide depth prediction, leveraging appearance differences across semantic classes. Using semantic guidance improves depth estimation, enabling state‑of‑the‑art accuracy with a simpler model by enforcing class‑based geometric constraints such as sky being far and ground being horizontal.

Abstract

We consider the problem of estimating the depth of each pixel in a scene from a single monocular image. Unlike traditional approaches, which attempt to map from appearance features to depth directly, we first perform a semantic segmentation of the scene and use the semantic labels to guide the 3D reconstruction. This approach provides several advantages: By knowing the semantic class of a pixel or region, depth and geometry constraints can be easily enforced (e.g., “sky” is far away and “ground” is horizontal). In addition, depth can be more readily predicted by measuring the difference in appearance with respect to a given semantic class. For example, a tree will have more uniform appearance in the distance than it does close up. Finally, the incorporation of semantic features allows us to achieve state-of-the-art results with a significantly simpler model than previous works.