DIB-R++: Learning to Predict Lighting and Material with a Hybrid\n Differentiable Renderer

Abstract

We consider the challenging problem of predicting intrinsic object properties\nfrom a single image by exploiting differentiable renderers. Many previous\nlearning-based approaches for inverse graphics adopt rasterization-based\nrenderers and assume naive lighting and material models, which often fail to\naccount for non-Lambertian, specular reflections commonly observed in the wild.\nIn this work, we propose DIBR++, a hybrid differentiable renderer which\nsupports these photorealistic effects by combining rasterization and\nray-tracing, taking the advantage of their respective strengths -- speed and\nrealism. Our renderer incorporates environmental lighting and spatially-varying\nmaterial models to efficiently approximate light transport, either through\ndirect estimation or via spherical basis functions. Compared to more advanced\nphysics-based differentiable renderers leveraging path tracing, DIBR++ is\nhighly performant due to its compact and expressive shading model, which\nenables easy integration with learning frameworks for geometry, reflectance and\nlighting prediction from a single image without requiring any ground-truth. We\nexperimentally demonstrate that our approach achieves superior material and\nlighting disentanglement on synthetic and real data compared to existing\nrasterization-based approaches and showcase several artistic applications\nincluding material editing and relighting.\n