2017 · 328 citations · 40 references
ComputingComputer ScienceComputer VisionMachine VisionVisual AttentionDeep LearningMotion AnalysisMotion EstimationScene UnderstandingSemantic SegmentationVideo AnalysisGated Recurrent Unit NetworksJoint Representation LearningMotion Representation LearningMotion SegmentationObject SegmentationSpatiotemporal Feature LearningVideo Representation LearningVideo Segmentation
The study proposes a method to segment moving objects in unconstrained videos. The authors design a two‑stream neural network that encodes spatial and temporal features and incorporates a convolutional gated recurrent memory module to build a visual memory of the video, enabling pixel‑wise object/background labeling without manual annotations. The method achieves state‑of‑the‑art performance, outperforming the best existing approach on DAVIS by almost 6 % and ranking highly on Freiburg‑Berkeley.
This paper addresses the task of segmenting moving objects in unconstrained videos. We introduce a novel two-stream neural network with an explicit memory module to achieve this. The two streams of the network encode spatial and temporal features in a video sequence respectively, while the memory module captures the evolution of objects over time. The module to build a “visual memory” in video, i.e., a joint representation of all the video frames, is realized with a convolutional recurrent unit learned from a small number of training video sequences. Given a video frame as input, our approach assigns each pixel an object or background label based on the learned spatio-temporal features as well as the "visual memory" specific to the video, acquired automatically without any manually-annotated frames. The visual memory is implemented with convolutional gated recurrent units, which allows to propagate spatial information over time. We evaluate our method extensively on two benchmarks, DAVIS and Freiburg-Berkeley motion segmentation datasets, and show state-of-the-art results. For example, our approach outperforms the top method on the DAVIS dataset by nearly 6%. We also provide an extensive ablative analysis to investigate the influence of each component in the proposed framework.
40
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · Nature · 1986 · 30.9K citations
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick et al. · arXiv (Cornell University) · 2015 · 18.2K citations · Full text