2023 · 10 citations · 11 references
ComputingComputer ScienceComputational LinguisticsComputer VisionNatural Language ProcessingAnnotationMultimodal InteractionNamed Entity RecognitionRelation ExtractionConstraintsCross-modal AlignmentCross-modal FusionCross-modal MatchingImage-text AlignmentText-to-image Generation
The challenge posed by multimodal named entity recognition (MNER) is mainly two-fold: (1) bridging the semantic gap between text and image and (2) matching the entity with its associated object in image. Existing methods fail to capture the implicit entity-object relations, due to the lack of corresponding annotation. In this paper, we propose a bidirectional generative alignment method named BGA-MNER to tackle these issues. Our BGA-MNER consists of image2text and text2image generation with respect to entity-salient content in two modalities. It jointly optimizes the bidirectional reconstruction objectives, leading to aligning the implicit entity-object relations under such direct and powerful constraints. Furthermore, image-text pairs usually contain unmatched components which are noisy for generation. A stage-refined context sampler is proposed to extract the matched cross-modal content for generation. Extensive experiments on two benchmarks demonstrate that our method achieves state-of-the-art performance without image input during inference.
11
On Information and Sufficiency
S. Kullback, R. A. Leibler · The Annals of Mathematical Statistics · 1951 · 20K citations · Full text
Neural Architectures for Named Entity Recognition
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian et al. · 2016 · 4.4K citations · Full text
Erik F. Tjong Kim Sang, Jorn Veenstra · 1999 · 459 citations · Full text
Visual Attention Model for Name Tagging in Multimodal Social Media
Di Lu, Leonardo Neves, Vı́tor Carvalho et al. · 2018 · 288 citations · Full text