2023 · 14 citations · 39 references
ComputingComputer ScienceComputer VisionMemoryVisual AttentionDeep LearningDepth EstimationMotion EstimationScene UnderstandingVideo AnalysisVisual MemoryAttentionDepth Map EstimationFuture Frame PredictionMonocular Depth EstimationSingle-image Depth EstimationTemporal Information IntegrationVideo Representation Learning
We propose MAMo, a novel memory and attention framework for monocular video depth estimation. MAMo can augment and improve any single-image depth estimation networks into video depth estimation models, enabling them to take advantage of the temporal information to predict more accurate depth. In MAMo, we augment model with memory which aids the depth prediction as the model streams through the video. Specifically, the memory stores learned visual and displacement tokens of the previous time instances. This allows the depth network to cross-reference relevant features from the past when predicting depth on the current frame. We introduce a novel scheme to continuously update the memory, optimizing it to keep tokens that correspond with both the past and the present visual information. We adopt attention-based approach to process memory features where we first learn the spatiotemporal relation among the resultant visual and displacement memory tokens using self-attention module. Further, the output features of self-attention are aggregated with the current visual features through cross-attention. The cross-attended features are finally given to a decoder to predict depth on the current frame. Through extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and DDAD, we show that MAMo consistently improves monocular depth estimation networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video depth estimation provides higher accuracy with lower latency, when comparing to SOTA cost-volume-based video depth models.
39
Are we ready for autonomous driving? The KITTI vision benchmark suite
Andreas Geiger, P Lenz, R. Urtasun · 2012 · 14.4K citations
SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers
Enze Xie, Wenhai Wang, Zhiding Yu et al. · arXiv (Cornell University) · 2021 · 3.3K citations · Full text
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Deqing Sun, Xiaodong Yang, Ming-Yu Liu et al. · 2018 · 2.7K citations