Proceedings of the 30th ACM International Conference on Multimedia · 2022 · 50 citations · 22 references
ComputingAutonomous SystemsComputer ScienceComputer VisionNatural Language ProcessingPath PlanningAutonomous NavigationNavigation PlanningRobot NavigationRobotic VisionScene UnderstandingVisual NavigationAutonomous Navigation ArchitecturesGlobal PlanningGoal-directed NavigationGoal-driven NavigationScene ExplorationScene Understanding For Navigation
Vision-language navigation is the task of directing an embodied agent to navigate in 3D scenes with natural language instructions. For the agent, inferring the long-term navigation target from visual-linguistic clues is crucial for reliable path planning, which, however, has rarely been studied before in literature. In this article, we propose a Target-Driven Structured Transformer Planner (TD-STP) for long-horizon goal-guided and room layout-aware navigation. Specifically, we devise an Imaginary Scene Tokenization mechanism for explicit estimation of the long-term target (even located in unexplored environments). In addition, we design a Structured Transformer Planner which elegantly incorporates the explored room layout into a neural attention architecture for structured and global planning. Experimental results demonstrate that our TD-STP substantially improves previous best methods' success rate by 2% and 5% on the test set of R2R and REVERIE benchmarks, respectively. Our code is available at https://github.com/YushengZhao/TD-STP.
22
Advances in neural information processing systems 7
Computers & Mathematics with Applications · 1996 · 14.4K citations
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Angela Dai, Manolis Savva, Maciej Halber et al. · 2017 · 4K citations
Gibson Env: Real-World Perception for Embodied Agents
Fei Xia, Amir Zamir, Zhiyang He et al. · 2018 · 715 citations