Information Processing Capacity of Dynamical Systems

TLDR

Many dynamical systems, both natural and artificial, are stimulated by time‑dependent external signals and process the information they contain. We demonstrate how to quantify the different modes of information processing in such systems and combine them to define a dynamical system’s computational capacity. Our theory blends concepts from reservoir computing, system modeling, stochastic processes, and functional analysis, and we illustrate it with numerical simulations of the logistic map, a recurrent neural network, and a two‑dimensional reaction‑diffusion system, revealing universal trade‑offs between computational non‑linearity and short‑term memory. The computational capacity is bounded by the number of linearly independent state variables, reaching this bound when the system satisfies the fading memory condition, and can be interpreted as the total number of linearly independent stimulus‑response functions the system can compute, as shown by the simulations that expose trade‑offs between non‑linearity and memory.

Abstract

Many dynamical systems, both natural and artificial, are stimulated by time dependent external signals, somehow processing the information contained therein. We demonstrate how to quantify the different modes in which information can be processed by such systems and combine them to define the computational capacity of a dynamical system. This is bounded by the number of linearly independent state variables of the dynamical system, equaling it if the system obeys the fading memory condition. It can be interpreted as the total number of linearly independent functions of its stimuli the system can compute. Our theory combines concepts from machine learning (reservoir computing), system modeling, stochastic processes and functional analysis. We illustrate our theory by numerical simulations for the logistic map, a recurrent neural network and a two-dimensional reaction diffusion system, uncovering universal trade-offs between the non-linearity of the computation and the system's short-term memory.