2023 · 27 citations · 26 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingInformation ExtractionNamed Entity RecognitionRelation ExtractionSelf-supervised LearningHierarchical FusionImage AlignmentImage-text AlignmentJoint Entity ExtractionJoint Entity RecognitionMultimodal Information ExtractionMultimodal Self-supervised LearningSelf-supervised Pretraining
How can we better extract entities and relations from text? Using multimodal extraction with images and text obtains more signals for entities and relations, and aligns them through graphs or hierarchical fusion, aiding in extraction. Despite attempts at various fusions, previous works have overlooked many unlabeled image-caption pairs, such as NewsCLIPing. This paper proposes innovative pre-training objectives for entity-object and relation-image alignment, extracting objects from images and aligning them with entity and relation prompts for soft pseudo-labels. These labels are used as self-supervised signals for pre-training, enhancing the ability to extract entities and relations. Experiments on three datasets show an average 3.41% F1 improvement over prior SOTA. Additionally, our method is orthogonal to previous multimodal fusions, and using it on prior SOTA fusions further improves 5.47% F1.
26
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Xuezhe Ma, Eduard Hovy · 2016 · 2.6K citations · Full text
Unbiased Scene Graph Generation From Biased Training
Kaihua Tang, Yulei Niu, Jianqiang Huang et al. · 2020 · 728 citations