Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

TLDR

Automatic 3D content creation has advanced rapidly thanks to large language models and image diffusion models, yet existing text‑to‑3D methods that use implicit scene representations couple geometry and appearance, limiting fine geometry recovery and photorealism. This work proposes Fantasia3D, a method for high‑quality text‑to‑3D content creation. Fantasia3D disentangles geometry and appearance by using a hybrid scene representation that feeds surface normals into an image diffusion model for geometry learning, and incorporates a spatially varying BRDF to learn surface material for photorealistic rendering. The disentangled framework is more compatible with graphics engines, enabling relighting, editing, and physical simulation, and experiments demonstrate its superiority over existing methods across various text‑to‑3D settings. Project page and source code available at https://fantasia3d.github.io/.

Abstract

Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.