2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 82 citations · 36 references
ComputingComputer ScienceRobotics EngineeringComputer VisionKnowledge CaptureKnowledge RepresentationGrasp Pose EstimationKnowledge AnnotationManipulation LearningMultimodal InteractionObject ManipulationObject RecognitionAffordance LearningAffordancesGraspHousehold ObjectsHuman Pose EstimationKnowledge Base ConstructionObject AffordanceObject Pose Estimation
Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them. In this work, we propose a multi-modal and rich-annotated knowledge repository, OakInk, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: Oak. Given the affordance, we record rich human interactions with 100 selected objects in Oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: Tink. The recorded and transferred hand-object interactions constitute the second knowledge base: Ink. As a result, OakInk contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark OakInk on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of OakInk: intent-based interaction generation and handover generation. Our dataset and source code are publicly available at www.oakink.net.
36
George A. Miller · Communications of the ACM · 1995 · 14.1K citations · Full text
ShapeNet: An Information-Rich 3D Model Repository
Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan et al. · arXiv (Cornell University) · 2015 · 2.4K citations · Full text
Learning the signatures of the human grasp using a scalable tactile glove
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li et al. · Nature · 2019 · 1.1K citations