Detecting Human-Object Interaction via Fabricated Compositional Learning

TLDR

Human‑Object Interaction detection is a core scene‑understanding task but is hindered by the open long‑tailed distribution of interactions, while humans can perceive rare or unseen HOIs compositionally. This work proposes Fabricated Compositional Learning (FCL) to tackle open long‑tailed HOI detection. FCL introduces an object fabricator that generates effective object representations, which are combined with verbs to synthesize new HOI samples, thereby producing large‑scale data for rare and unseen categories. On the HICO‑DET benchmark, FCL markedly improves state‑of‑the‑art performance for imbalanced and unseen HOI categories. Code is available at https://github.com/zhihou7/HOI-CL.

Abstract

Human-Object Interaction (HOI) detection, inferring the relationships between human and objects from images/videos, is a fundamental task for high-level scene understanding. However, HOI detection usually suffers from the open long-tailed nature of interactions with objects, while human has extremely powerful compositional perception ability to cognize rare or unseen HOI samples. Inspired by this, we devise a novel HOI compositional learning framework, termed as Fabricated Compositional Learning (FCL), to address the problem of open long-tailed HOI detection. Specifically, we introduce an object fabricator to generate effective object representations, and then combine verbs and fabricated objects to compose new HOI samples. With the proposed object fabricator, we are able to generate large-scale HOI samples for rare and unseen categories to alleviate the open long-tailed issues in HOI detection. Extensive experiments on the most popular HOI detection dataset, HICO-DET, demonstrate the effectiveness of the proposed method for imbalanced HOI detection and significantly improve the state-of-the-art performance on rare and unseen HOI categories. Code is available at https://github.com/zhihou7/HOI-CL.