Information Fusion · 2023 · 82 citations · 32 references
ComputingComputer ScienceComputer VisionDigital RenderingRendering3D RenderingObject RecognitionRendering PipelineScene Understanding3D Scene UnderstandingLarge Language ModelsLarge-scale Natural Language ProcessingLatent Space ExplorationMultimodal FusionMultimodal Representation LearningSemantic RelationshipsUnseen Category Handling
Zero-shot 3D shape understanding aims to recognize “unseen” 3D categories that are not present in training data. Recently, Contrastive Language-Image Pre-training (CLIP) has shown promising open-world performance in zero-shot 3D shape understanding tasks by information fusion among language and 3D modality. It first renders 3D objects into multiple 2D image views and then learns to understand the semantic relationships between the textual descriptions and images, enabling the model to generalize to new and unseen categories. However, existing studies in zero-shot 3D shape understanding rely on predefined rendering parameters, resulting in repetitive, redundant, and low-quality views. This limitation hinders the model’s ability to fully comprehend 3D shapes and adversely impacts the text-image fusion in a shared latent space. To this end, we propose a novel approach called Differentiable rendering-based multi-view Image-Language Fusion (DILF) for zero-shot 3D shape understanding. Specifically, DILF leverages large-scale language models (LLMs) to generate textual prompts enriched with 3D semantics and designs a differentiable renderer with learnable rendering parameters to produce representative multi-view images. These rendering parameters can be iteratively updated using a text-image fusion loss, which aids in parameters’ regression, allowing the model to determine the optimal viewpoint positions for each 3D object. Then a group-view mechanism is introduced to model interdependencies across views, enabling efficient information fusion to achieve a more comprehensive 3D shape understanding. Experimental results can demonstrate that DILF outperforms state-of-the-art methods for zero-shot 3D classification while maintaining competitive performance for standard 3D classification. The code is available at https://github.com/yuzaiyang123/DILP.
32
Proceedings of IEEE International Conference on Computer Vision
1995 · 12.9K citations
Automatic differentiation in PyTorch
Adam Paszke, Sam Gross, Soumith Chintala et al. · 2017 · 11.1K citations
ShapeNet: An Information-Rich 3D Model Repository
Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan et al. · arXiv (Cornell University) · 2015 · 2.4K citations · Full text
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
Guocheng Qian, Yuchen Li, Houwen Peng et al. · arXiv (Cornell University) · 2022 · 415 citations · Full text