Proceedings of the AAAI Conference on Artificial Intelligence · 2022 · 120 citations · 31 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningDocument Image AnalysisDocument LayoutInformation ExtractionLanguage ModelingNatural Language UnderstandingDocument UnderstandingDocument-level Information ExtractionImage-text AlignmentKey Concept ExtractionKey ExtractionKey Information ExtractionMaskingText-to-image Alignment
Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks--(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples--and demonstrates the superiority of BROS over previous methods.
31
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang et al. · 2019 · 3.1K citations · Full text