Complexity · 1996 · 11 citations · 0 references
Natural SciencesComputer ScienceMathematicsInformation TheoryMachine LearningProbability TheoryAlgorithmic ComplexityEntropyInformation MeasuresAlgorithmic Information TheoryComplexityInformation ComplexityInformation EntropyMeasure-theoretic EntropyOptimization Of Complex SystemsShannon Entropy
Effective complexity is the length of a compact description of an entity’s regularities, while total information adds an entropy term describing its random aspects. The article defines an information measure and demonstrates how entropy, Shannon information, and algorithmic information content can be combined to address characterization, inference, and learning in complex systems. Mathematical definitions of effective complexity and total information are provided, and the authors discuss applications of these measures. The study shows that selecting the set of regularities that minimizes total information and then effective complexity yields an observer‑independent effective complexity. © 1996 John Wiley & Sons, Inc.
This article defines the concept of an information measure and shows how common information measures such as entropy, Shannon information, and algorithmic information content can be combined to solve problems of characterization, inference, and learning for complex systems. Particularly useful quantities are the effective complexity, which is roughly the length of a compact description of the identified regularities of an entity, and total information, which is effective complexity plus an entropy term that measures the information required to describe the random aspects of the entity. Mathematical definitions are given for both quantities and some applications are discussed. In particular, it is pointed out that if one compares different sets of identified regularities of an entity, the ‘best’ set minimizes the total information, and then, subject to that constraint, minimizes the effective complexity; the resulting effective complexity is then in many respects independent of the observer. © 1996 John Wiley & Sons, Inc.