A system for image-text relations in new (and old) media

TLDR

Image–text relations are framed by two dimensions: the relative status of images and texts, and their logico–semantic connections. The article introduces a generalized system for modeling image–text relations across multimodal genres. Each image–text pair is characterized by selected features, with units identified and logico–semantic and status relations specified for both human analysts and machines, and two application scenarios are illustrated. The system enables distinguishing image–text relations in genuinely new media from those in older media.

Abstract

This article presents a generalized system of image–text relations which applies to different genres of multimodal discourse in which images and texts co-occur. It combines two kinds of relations – the relative status of images and text, and how they relate to one another in terms of logico–semantics. Every instance of an image–text combination in the data sample is described by a selection of features from the system. The units of images and text between which the relations obtain are identified and the realizations of the logico–semantic and status relations are specified, both for the human analyst and a machine. Two application scenarios are discussed. The system should be useful for distinguishing between image–text relations for (genuinely) new and old media.