Optimization of directed information and relations to filtering theory

Abstract

In this paper we generalize the relation between nonanticipative Rate Distortion Function (RDF) and filtering theory, to processes which are affected by the reproduction process, by utilizing the topology of weak convergence of probability measures. Specifically, this generalization is established via an optimization on the space of conditional distributions of the so-called directed information, subject to a fidelity constraint. Existence of the optimal reproduction distribution of the general nonanticipative RDF is shown, while the closed form expression of the optimal reproduction distribution is obtained for nonstationary processes. The expression of the optimal reproduction conditional distribution is recursively computed, backward in time. Finally, the realization procedure of the general nonanticipative RDF which is equivalent to joint-source channel matching for symbol-by-symbol transmission is described.