Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition

TLDR

Interpreting human–object interactions in images and videos is challenging, requiring scene understanding, human motion analysis, object recognition, and observation of their effects, and recognition improves when these perceptual tasks are integrated rather than performed independently, unlike prior methods that rely solely on static shape or motion cues. The study proposes a Bayesian framework that integrates multiple perceptual tasks to model human–object interactions. The method applies spatial and functional constraints to each perceptual element within a Bayesian model for coherent semantic interpretation. The constraints enable accurate object and action recognition even when appearances are ambiguous and allow action inference from static images without motion data.

Abstract

Interpretation of images and videos containing humans interacting with different objects is a daunting task. It involves understanding scene/event, analyzing human movements, recognizing manipulable objects, and observing the effect of the human movement on those objects. While each of these perceptual tasks can be conducted independently, recognition rate improves when interactions between them are considered. Motivated by psychological studies of human perception, we present a Bayesian approach which integrates various perceptual tasks involved in understanding human-object interactions. Previous approaches to object and action recognition rely on static shape/appearance feature matching and motion analysis, respectively. Our approach goes beyond these traditional approaches and applies spatial and functional constraints on each of the perceptual elements for coherent semantic interpretation. Such constraints allow us to recognize objects and actions when the appearances are not discriminative enough. We also demonstrate the use of such constraints in recognition of actions from static images without using any motion information.