Parsing Natural Scenes and Natural Language with Recursive Neural Networks

TLDR

Recursive structure is common in natural scene images and natural language sentences, and uncovering it enables identification of constituent units and their interactions. The authors introduce a max‑margin structure‑prediction architecture based on recursive neural networks to recover recursive structure in complex scene images and natural language sentences. The architecture uses recursive neural networks for structure prediction, serving as a competitive syntactic parser for Penn Treebank sentences and outperforming alternative methods for semantic scene segmentation, annotation, and classification. The algorithm attains 78.1 % accuracy on the Stanford background dataset and improves scene classification by 4 % over Gist descriptors.

Abstract

Recursive structure is commonly found in the inputs of different modalities such as natural scene images or natural language sentences. Discovering this recursive structure helps us to not only identify the units that an image or sentence contains but also how they interact to form a whole. We introduce a max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences. The same algorithm can be used both to provide a competitive syntactic parser for natural language sentences from the Penn Treebank and to outperform alternative approaches for semantic scene segmentation, annotation and classification. For segmentation and annotation our algorithm obtains a new level of state-of-the-art performance on the Stanford background dataset (78.1%). The features from the image parse tree outperform Gist descriptors for scene classification by 4%.