2019 · 18 citations · 20 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDialogue SystemsLanguage ModelingLanguage ModelsPragmatic InferencePragmatic ProcessingSpeech ActsAccuracyDistractor GenerationDistractor ProcessingObject NamingObjectsPragmatic UnderstandingReferring Expression UnderstandingZero-shot Image Labeling
Zero-shot learning in Language & Vision is the task of correctly labelling (or naming) objects of novel categories. Another strand of work in L&V aims at pragmatically informative rather than "correct" object descriptions, e.g. in reference games. We combine these lines of research and model zero-shot reference games, where a speaker needs to successfully refer to a novel object in an image. Inspired by models of "rational speech acts", we extend a neural generator to become a pragmatic speaker reasoning about uncertain object categories. As a result of this reasoning, the generator produces fewer nouns and names of distractor categories as compared to a literal speaker. We show that this conversational strategy for dealing with novel objects often improves communicative success, in terms of resolution accuracy of an automatic listener.
20
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio et al. · 2015 · 6.4K citations
Referring as a collaborative process
Herbert H. Clark, Deanna Wilkes-Gibbs · Cognition · 1986 · 2.4K citations
Management, Organization Studies, Business Process Management +11