2024 · 10 citations · 52 references
ArchitectureComputer ScienceComputer GraphicsReal-time RenderingRendering3D RenderingPhysically Based RenderingReal-time Rendering For Virtual RealityRendering PipelineVirtual RealityVirtual WorldsNeural Radiance FieldsRadianceReal-time Environment RenderingRealistic SceneryRender Engines
Creating high-quality and interactive virtual environments, such as games and simulators, often involves complex and costly manual modeling processes. In this paper, we present Video2Game, a novel approach that automatically converts videos of real-world scenes into realistic and interactive game environments. At the heart of our system are three core components: (i) a neural radiance fields (NeRF) module that effectively captures the geometry and visual appearance of the scene; (ii) a mesh module that distills the knowledge from NeRF for faster rendering; and (iii) a physics module that models the interactions and physical dynamics among the objects. By following the carefully designed pipeline, one can construct an interactable and actionable digital replica of the real world. We benchmark our system on both indoor and large-scale outdoor scenes. We show that we can not only produce highly-realistic renderings in real-time, but also build interactive games on top.
52
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros et al. · 2018 · 12.4K citations
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler et al. · ACM Transactions on Graphics · 2023 · 4.5K citations · Full text
MuJoCo: A physics engine for model-based control
Emanuel Todorov, Tom Erez, Yuval Tassa · 2012 · 4.4K citations
Engineering, Mechanical Engineering, Robotics Engineering +17
Marc Levoy, Pat Hanrahan · 1996 · 3.7K citations · Full text