Mutual Information and Minimum Mean-Square Error in Gaussian Channels

TLDR

This paper studies arbitrarily distributed finite‑power input signals observed through an additive Gaussian noise channel. The study introduces a formula linking input‑output mutual information to the minimum mean‑square error (MMSE) achievable by optimal estimation of the input given the output. This formula is derived for arbitrarily distributed finite‑power input signals transmitted over an additive Gaussian noise channel. The derivative of mutual information with respect to SNR equals half the MMSE for any input statistics, a relationship that holds for scalar and vector, discrete‑time and continuous‑time cases, and implies that for finite‑power inputs the causal filtering MMSE at a given SNR equals the average noncausal smoothing MMSE over uniformly distributed SNRs.

Abstract

This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.