IEEE Transactions on Circuits and Systems for Video Technology · 2024 · 39 citations · 35 references
ComputingVision ScienceComputer VisionDigital RenderingRendering3D Reconstruction3D RenderingMotion EstimationOptical FlowRendering PipelineScene ReconstructionDense Optical FlowDynamic ReconstructionDynamic Scene ReconstructionDynamic Scene RepresentationFlow AugmentationMotion-aware ReconstructionOptical Flow EstimationTransient Deformation
3D Gaussian Splatting (3DGS) has become an emerging tool for dynamic scene reconstruction. However, existing methods mainly focus on developing various strategies to extend static 3DGS into a time-variant representation, while overlooking the rich motion information implicitly carried by 2D observations, thus suffering from performance degradation and model redundancy. To address the above problem, we propose a novel motion-aware enhancement framework for dynamic scene reconstruction, which mines useful motion cues from optical flow to improve different paradigms of dynamic 3DGS. Specifically, we first step beyond the vanilla render-based cross-dimensional supervision that suffers from ambiguity and instability, and establish a more robust and effective dense correspondence between 3D Gaussian movements and pixel-level flows. Then a novel flow augmentation method is introduced with additional insights into uncertainty and loss collaboration. Furthermore, for the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed. We conduct extensive experiments on both multi-view and monocular scenes to verify the merits of our work. Compared with the baselines, our method shows significant superiority in both rendering quality and efficiency. The code will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jasongzy/MAGS</uri>.
35
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros et al. · 2018 · 12.4K citations
Alexander M. Kirillov, Eric Mintun, Nikhila Ravi et al. · 2023 · 9.1K citations
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler et al. · ACM Transactions on Graphics · 2023 · 4.5K citations · Full text
A Database and Evaluation Methodology for Optical Flow
Simon Baker, Daniel Scharstein, John Lewis et al. · International Journal of Computer Vision · 2010 · 2.2K citations · Full text