Neural Rays for Occlusion-aware Image-based Rendering

TLDR

Recent radiance‑field methods synthesize novel views from image features but struggle with occlusions, as invisible 3D points introduce inconsistent features that degrade rendering quality. This work introduces Neural Ray (NeuRay) to predict the visibility of 3D points for each input view, thereby mitigating occlusion effects. NeuRay jointly learns a visibility map and a radiance field, and employs a consistency loss during scene‑specific fine‑tuning to refine visibility predictions. The resulting model attains state‑of‑the‑art performance on novel view synthesis, generalizing to unseen scenes and surpassing per‑scene optimization after fine‑tuning. Project page: https://liuyuan-pal.github.io/NeuRay/.

Abstract

We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms perscene optimization methods after finetuning. Project page:https://liuyuan-pal.github.io/NeuRay/