The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and\n Sentences From Natural Supervision

Abstract

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns\nvisual concepts, words, and semantic parsing of sentences without explicit\nsupervision on any of them; instead, our model learns by simply looking at\nimages and reading paired questions and answers. Our model builds an\nobject-based scene representation and translates sentences into executable,\nsymbolic programs. To bridge the learning of two modules, we use a\nneuro-symbolic reasoning module that executes these programs on the latent\nscene representation. Analogical to human concept learning, the perception\nmodule learns visual concepts based on the language description of the object\nbeing referred to. Meanwhile, the learned visual concepts facilitate learning\nnew words and parsing new sentences. We use curriculum learning to guide the\nsearching over the large compositional space of images and language. Extensive\nexperiments demonstrate the accuracy and efficiency of our model on learning\nvisual concepts, word representations, and semantic parsing of sentences.\nFurther, our method allows easy generalization to new object attributes,\ncompositions, language concepts, scenes and questions, and even new program\ndomains. It also empowers applications including visual question answering and\nbidirectional image-text retrieval.\n