2024 · 22 citations · 26 references
ComputingComputer ScienceRobotics EngineeringComputer VisionRobot PerceptionRobot Vision3D ReconstructionMappingRobotic VisionScene ReconstructionScene UnderstandingSemantic Segmentation3D Scene Representation3D Scene UnderstandingConfidence MapsRegion-based RepresentationRgb-d ReconstructionScene RepresentationSemantic ComprehensionSemantic Scene Understanding
Precise 3D environmental mapping with semantics is essential in robotics. Existing methods often rely on pre-defined concepts during training or are time-intensive when generating semantic maps. This paper presents Open-Fusion, an approach for real-time open-vocabulary 3D mapping and queryable scene representation using RGB-D data. Open-Fusion harnesses the power of a pretrained vision-language foundation model (VLFM) for open-set semantic comprehension and employs the Truncated Signed Distance Function (TSDF) for swift 3D scene reconstruction. By leveraging the VLFM, we extract region-based embeddings and their associated confidence maps. These are then integrated with the 3D knowledge from TSDF using an enhanced Hungarian-based feature-matching mechanism. In particular, Open-Fusion delivers outstanding annotation-free 3D segmentation for open vocabulary query without the need for additional 3D training. Benchmark tests on the ScanNet dataset against leading zero-shot methods highlight Open-Fusion’s superiority. Furthermore, it seamlessly combines the strengths of region-based VLFM and TSDF, facilitating real-time 3D scene comprehension that includes object concepts and open-world semantics. We encourage the readers to view the demos on our project page: https://uark-aicv.github.io/OpenFusion
26
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Nils Reimers, Iryna Gurevych · 2019 · 10.9K citations · Full text
Alexander M. Kirillov, Eric Mintun, Nikhila Ravi et al. · 2023 · 9.1K citations
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Angela Dai, Manolis Savva, Maciej Halber et al. · 2017 · 4K citations
SUN RGB-D: A RGB-D scene understanding benchmark suite
Shuran Song, Samuel P. Lichtenberg, Jianxiong Xiao · 2015 · 1.9K citations