IEEE Transactions on Visualization and Computer Graphics · 2023 · 12 citations · 42 references
EngineeringVision ScienceComputer VisionMachine VisionProjection Mapping3D RenderingImage SynthesisProjection-based Augmented RealityReflectance ModelingScene ReconstructionBrdf ModelingBrdf ParameterizationMulti-view SynthesisPhoto-realistic RenderingProjector CalibrationReflectance Field RepresentationView-dependent ProjectionVisual Scene Understanding
We introduce a high resolution spatially adaptive light source, or a projector, into a neural reflectance field that allows to both calibrate the projector and photo realistic light editing. The projected texture is fully differentiable with respect to all scene parameters, and can be optimized to yield a desired appearance suitable for applications in augmented reality and projection mapping. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. Using an analytical BRDF model and carefully selected projection patterns, our acquisition process is simple and intuitive, featuring a fixed uncalibrated projected and a handheld camera with a co-located light source. As we demonstrate, the virtual projector incorporated into the pipeline improves scene understanding and enables various projection mapping applications, alleviating the need for time consuming calibration steps performed in a traditional setting per view or projector location. In addition to enabling novel viewpoint synthesis, we demonstrate state-of-the-art performance projector compensation for novel viewpoints, improvement over the baselines in material and scene reconstruction, and three simply implemented scenarios where projection image optimization is performed, including the use of a 2D generative model to consistently dictate scene appearance from multiple viewpoints. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.
42
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik et al. · Communications of the ACM · 2021 · 5.8K citations · Full text
Structure-from-Motion Revisited
Johannes L. Schönberger, Jan‐Michael Frahm · 2016 · 5.8K citations
Denoising Diffusion Probabilistic Models
arXiv (Cornell University) · 2020 · 5.6K citations · Full text
U2-Net: Going deeper with nested U-structure for salient object detection
Xuebin Qin, Zichen Zhang, Chenyang Huang et al. · Pattern Recognition · 2020 · 2.3K citations · Full text