Position-Augmented Transformers with Entity-Aligned Mesh for TextVQA

Abstract

In addition to visual components, many images usually contain valuable text information, which is essential for understanding the scene. Thus, we study the TextVQA task that requires reading texts in images to answer corresponding questions. However, most of previous works utilize sophisticated graph structure and manually crafted features to model the position relationship between visual entities and texts in images. And traditional multimodal transformers cannot effectively capture relative position information and original image features. To address these issues in an intuitive but effective way, we propose a novel model, position-augmented transformers with entity-aligned mesh, for the TextVQA task. Different from traditional attention mechanism in transformers, we explicitly introduce continuous relative position information of objects and OCR tokens without complex rules. Furthermore, we replace the complicated graph structure with intuitive entity-aligned mesh according to perspective mapping. In this mesh, the information of discrete entities and image patches at different positions can interact with each other. Extensive experiments on two benchmark datasets (TextVQA and ST-VQA) show that our proposed model is superior to several state-of-the-art methods.