Proceedings of the AAAI Conference on Artificial Intelligence · 2021 · 69 citations · 28 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingComputer Vision In EducationDocument Image AnalysisInformation ExtractionObject DetectionObject RecognitionPattern RecognitionDocument UnderstandingFeature FusionInformation Extraction From Structured DocumentsMultimodal Feature FusionMultimodal FusionSemantic Context CluesText DetectionVisual Feature Extraction
Visual Information Extraction has attracted attention for applications such as document understanding, but prior work decouples text spotting and extraction, ignoring their interdependence. This paper proposes a robust end‑to‑end Visual Information Extraction System that simultaneously detects, recognizes, and extracts information from a single document image, and introduces the EPHOIE dataset, the first Chinese benchmark for both tasks. The system fuses visual and semantic features from text spotting into an extraction branch for multimodal fusion, while the extraction branch supplies high‑level clues to improve text spotting, and the EPHOIE dataset contains 1,494 exam‑paper images with 15,771 Chinese text instances. VIES outperforms state‑of‑the‑art methods, achieving a 9.01 % F‑score improvement on the SROIE dataset in end‑to‑end evaluation.
Visual Information Extraction (VIE) has attracted considerable attention recently owing to its various advanced applications such as document understanding, automatic marking and intelligent education. Most existing works decoupled this problem into several independent sub-tasks of text spotting (text detection and recognition) and information extraction, which completely ignored the high correlation among them during optimization. In this paper, we propose a robust Visual Information Extraction System (VIES) towards real-world scenarios, which is an unified end-to-end trainable framework for simultaneous text detection, recognition and information extraction by taking a single document image as input and outputting the structured information. Specifically, the information extraction branch collects abundant visual and semantic representations from text spotting for multimodal feature fusion and conversely, provides higher-level semantic clues to contribute to the optimization of text spotting. Moreover, regarding the shortage of public benchmarks, we construct a fully-annotated dataset called EPHOIE (https://github.com/HCIILAB/EPHOIE), which is the first Chinese benchmark for both text spotting and visual information extraction. EPHOIE consists of 1,494 images of examination paper head with complex layouts and background, including a total of 15,771 Chinese handwritten or printed text instances. Compared with the state-of-the-art methods, our VIES shows significant superior performance on the EPHOIE dataset and achieves a 9.01% F-score gain on the widely used SROIE dataset under the end-to-end scenario.
28
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Kaiming He, Georgia Gkioxari, Piotr Dollár et al. · 2017 · 28.8K citations
Feature Pyramid Networks for Object Detection
Tsung-Yi Lin, Piotr Dollár, Ross Girshick et al. · 2017 · 28.7K citations