2021 IEEE/CVF International Conference on Computer Vision (ICCV) · 2021 · 20 citations · 35 references
ComputingComputer ScienceComputer GraphicsComputer VisionMotion PlanningImage SynthesisMotion EstimationVideo AnalysisFramesMotion InterpolationVideo Frame InterpolationVideo Generation
Video frame interpolation aims to synthesize accurate intermediate frames given a low-frame-rate video. While the quality of the generated frames is increasingly getting better, state-of-the-art models have become more and more computationally expensive. However, local regions with small or no motion can be easily interpolated with simple models and do not require such heavy compute, whereas some regions may not be correct even after inference through a large model. Thus, we propose an effective framework that assigns varying amounts of computation for different regions. Our dynamic architecture first calculates the approximate motion magnitude to use as a proxy for the difficulty levels for each region, and decides the depth of the model and the scale of the input. Experimental results show that static regions pass through a smaller number of layers, while the regions with larger motion are downscaled for better motion reasoning. In doing so, we demonstrate that the proposed framework can significantly reduce the computation cost (FLOPs) while maintaining the performance, often up to 50% when interpolating a 2K resolution video.
35
Automatic differentiation in PyTorch
Adam Paszke, Sam Gross, Soumith Chintala et al. · 2017 · 11.1K citations
FlowNet: Learning Optical Flow with Convolutional Networks
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg et al. · 2015 · 4.2K citations
Categorical Reparameterization with Gumbel-Softmax
Eric Jang, Shixiang Gu, Ben Poole · arXiv (Cornell University) · 2016 · 3.2K citations · Full text
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Deqing Sun, Xiaodong Yang, Ming-Yu Liu et al. · 2018 · 2.7K citations
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Bichen Wu, Kurt Keutzer, Xiaoliang Dai et al. · 2019 · 1.3K citations