Multimodal Distributional Semantics

TLDR

Distributional semantic models compute word representations from co‑occurrence patterns in text and have proven effective for many tasks, yet they rely solely on textual data and miss the rich perceptual grounding of human semantic knowledge. The authors aim to overcome this limitation by incorporating computer vision–derived visual words into distributional models, thereby extending word representations to include co‑occurrence with image content. They develop a flexible architecture that integrates text‑based and image‑based distributional information, using vision techniques to automatically identify discrete visual words in images. Empirical evaluations demonstrate that the integrated model outperforms the purely text‑based approach and supplies complementary semantic information.

Abstract

Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete “visual words” in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.