SUPP: understanding moving picture patterns based on linguistic knowledge

Abstract

This paper presents a system for understanding moving picture patterns based on linguistic knowledge. In order to deal with the difference of knowledge level between picture pattern and natural language, a hierarchical structure is adopted for the representation of knowledge. At a certain level knowledge data is organized as frame centred around the concept of verb. The concepts of approximately five thousand Japanese verbs were systematically classified to construct such a knowledge system. The program reads a picture pattern sequence. Pairing a picture pattern with the successive one in the sequence, it recognizes primitive pictures in the pair. Then, checking dynamic relations among primitive pictures, it organizes the relations into verb-centred events . It makes inferences about similarity between two concepts of verbs . Finally it describes each event in Japanese and English.