Proceedings of the AAAI Conference on Artificial Intelligence · 2018 · 28 citations · 36 references
ComputingComputer ScienceComputational LinguisticsComputer VisionNatural Language ProcessingLanguage ModelingMultimodal CorporaMultimodal InteractionNatural Language UnderstandingVisual SemanticsWord EmbeddingsContextual Word EmbeddingsContextual Word RepresentationJoint Feature EmbeddingMulti-modal Representation LearningMultimodal EmbeddingMultimodal Representation LearningSkip-gram ModelVisual Context Effects
Representing the semantics of words is a long-standing problem for the natural language processing community. Most methods compute word semantics given their textual context in large corpora. More recently, researchers attempted to integrate perceptual and visual features. Most of these works consider the visual appearance of objects to enhance word representations but they ignore the visual environment and context in which objects appear. We propose to unify text-based techniques with vision-based techniques by simultaneously leveraging textual and visual context to learn multimodal word embeddings. We explore various choices for what can serve as a visual context and present an end-to-end method to integrate visual context elements in a multimodal skip-gram model. We provide experiments and extensive analysis of the obtained results.
36
Rethinking the Inception Architecture for Computer Vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe et al. · 2016 · 30.9K citations
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text