A Sensory-Motor Language for Human Activity Understanding

Abstract

We have empirically discovered that the space of human actions has a linguistic framework. This is a sensory-motor space consisting of the evolution of the joint angles of the human body in movement. The space of human activity has its own phonemes, morphemes, and sentences. We present a human activity language (HAL) for symbolic non-arbitrary representation of visual and motor information. In phonology, we define atomic segments (kinetemes) that are used to compose human activity. We introduce the concept of a kinetological system and propose five basic properties for such a system: compactness, view-invariance, reproducibility, selectivity, and reconstructivity. In morphology, we extend sequential language learning to incorporate associative learning with our parallel learning approach. Parallel learning is effective in identifying the kinetemes and active joints in a particular action. In syntax, we suggest four lexical categories for our human activity language (noun, verb, adjective, and adverb). These categories are combined into sentences through syntax for human movement