A general framework for adaptive processing of data structures

TLDR

Symbolic processing requires structured information, whereas connectionist models usually assume simple structures such as arrays or sequences; the proposed framework extends recurrent neural networks and hidden Markov models to acyclic graphs. The authors aim to unify adaptive models such as neural nets and belief nets to process structured information by studying supervised learning as transductions between structured input and output spaces with recursive hidden state representations. The framework models relations among variables with directed acyclic graphs and represents supervised transductions through recursive networks—cyclic graphs with delay elements—that are unfolded into encoding networks, allowing inference and learning algorithms from neural networks or graphical models to be applied. The representation enables integration of symbolic and subsymbolic data and allows inference and learning algorithms from neural networks or probabilistic graphical models to be directly applied.

Abstract

A structured organization of information is typically required by symbolic processing. On the other hand, most connectionist models assume that data are organized according to relatively poor structures, like arrays or sequences. The framework described in this paper is an attempt to unify adaptive models like artificial neural nets and belief nets for the problem of processing structured information. In particular, relations between data variables are expressed by directed acyclic graphs, where both numerical and categorical values coexist. The general framework proposed in this paper can be regarded as an extension of both recurrent neural networks and hidden Markov models to the case of acyclic graphs. In particular we study the supervised learning problem as the problem of learning transductions from an input structured space to an output structured space, where transductions are assumed to admit a recursive hidden state-space representation. We introduce a graphical formalism for representing this class of adaptive transductions by means of recursive networks, i.e., cyclic graphs where nodes are labeled by variables and edges are labeled by generalized delay elements. This representation makes it possible to incorporate the symbolic and subsymbolic nature of data. Structures are processed by unfolding the recursive network into an acyclic graph called encoding network. In so doing, inference and learning algorithms can be easily inherited from the corresponding algorithms for artificial neural networks or probabilistic graphical model.