Unit 6: All

Limitations of Shannon
Entropy
and Computable
Measures
Dr. Hector Zenil

To illustrate the limitations of entropy
we created a very simple graph:
1. Let 1 → 2 be a starting graph G connecting a
node with label 1 to a node with label 2. If a node
with label n has degree n, we call it a core node;
otherwise, we call it a supportive node.
2. Iteratively add a node n + 1 to G such that the
number of core nodes in G is maximised.

Algorithm:
The degree sequence d of the labeled nodes
d = 1,2, 3, 4, . . . , n
is the Champernowne constant in base 10, a transcendental real
whose decimal expansion is Borel normal.

Algorithm:
The sequence of number of edges is a function of core and supportive nodes, defined by
[1/r] + [2/r] +···+ [n/r],
where r = (1 + √5)/2 is the golden ratio and [· ] the floor
function (sequence A183136 in the OEIS), whose values are
1, 2, 4, 7, 10, 14, 18, 23, 29, 35, 42, 50, 58, 67, 76, 86, 97, 108,
120, 132, 145,....

Algorithm:
From iteration 1 to
iteration 3:

Maximum Entropy (MaxEnt)
Model Principle
The principle of maximum entropy states that the
probability distribution which best represents the
current state of knowledge is the one with largest
entropy…
E.T. Jaynes

Computing the
Uncomputable:
The Coding Theorem
Method
Dr. Hector Zenil

K(s) = min{ |p| | U(p) = s}
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}

Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s

Constructor
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s

Compatible
with the
observation
Constructor
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s

Algorithmic Data Analytics, Small Data Matters and Correlation
versus Causation. In Computability of the World? Philosophy and
Science in the Age of Big Data), Springer Verlag, 2017
suggestive
methods
Coding Theorem Method
(CTM)

Algorithmic Data Analytics, Small Data Matters and Correlation
versus Causation. In Computability of the World? Philosophy and
Science in the Age of Big Data), Springer Verlag, 2017
Coding Theorem Method
(CTM)
suggestive
methods

The Coding Theorem Method
(CTM)

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096223

How it Looks Like?

So what about the constant in
the Invariance Theorem?

Shannon Entropy and
Algorithmic Complexity
Joining Forces:
The Block Decomposition
Method (BDM)
Dr. Hector Zenil

Divide and Conquer
sample
sample
sample
sample
sample
If any part of the whole system (samples) is of high m(x) and low K(x), then that part can
be generated by mechanistic/algorithmic means and thus is causal. The lower BDM the
more causal.
Rube
Goldberg
machine

Block Decomposition Method
(BDM)
sample
sample
sample
sample
If any part of the whole system (samples) is of high m(x) and low K(x), then that part can
be generated by mechanistic/algorithmic means and thus is causal. The lower BDM the
more causal.
Rube
Goldberg
machine

Formally:
(BDM)
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

(BDM)

(BDM)
For example:
If 2 strings are 10111 and 10111, a tighter upper bound of their
algorithmic complexity K is not:
|K(U(p)=10111)| + |K(U(p)=10111)| but
|K(U(p)=10111 2 times)|

Hybrid Measure:
Shannon Entropy +
Local Algorithmic Complexity
Because clearly:
|K(U(p)=10111)| + |K(U(p)=10111)| >|K(U(p)=10111 2 times)|

Hybrid Measure:
Shannon Entropy +
Local Algorithmic Complexity

e.g. These 2 strings are of high Entropy (and high Entropy rate)
but low K:
010101110100, 011011010010
Gaining Causation Power

Examples

CTM and BDM conform with the
theoretical expectation

Example of a generating
mechanism/model

Algorithmic Sequence Model
Sequential aggregation of small models produce a model able to explain larger
data. TMs: 2205 <> 1351 <> 1447 <> 599063 can produce:
S = 00000101111111111111111111110
+ +
+

The Online Algorithmic
Complexity Calculator
complexitycalculator.com

Version history and future
of the OACC
We keep expanding the calculator to wider
horizons of methodological and numerical capabilities:
• Version 1: Estimations of K for short binary strings
• Version 2: Expanded CTM and BDM capabilities for non-binary
strings and arrays
• Version 2.5: Estimations of Bennett's logical depth based on
CTM
• Version 3 (current version): Algorithmic Information dynamics
of strings, arrays and networks
• Version 4: Algorithmic complexity of models, algorithmic feature
selection, algorithmic dimensionality reduction and model
generator. (BETA version ready!)
complexitycalculator.com

We have seen how important AIT and
the role of Computability theory is in
the challenge of causality in science in
the 4th revolution of causal discover
towards model-driven approaches
https://www.youtube.com/watch?v=BEaXyDS_1Cw&t=30s

Graph and Tensor
Algorithmic
Complexity
Dr. Hector Zenil

A
B
C
Random versus simple 2D objects

Boundary Conditions

Scatterplot of 2-dimensional CTM versus
1-dimensional CTM
https://peerj.com/articles/cs-23/

ECAs sorted by
BDM

BDM on CAs
compared to
compression:

Algorithmic
Information Dynamics
Dr. Hector Zenil

The 2D CA Game of Life (GoL)
Density diagram showing the
persistence of structures:
The Game of Life

The Game of Life Space-time
evolution
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

GoL’s glider
(a ‘moving’ particle)
A piece of information moving through the grid from one side to another

Pattern algorithmic dynamics
GoL’s glider algorithmic dynamics
The amount of information is preserved,
neither lost or increased

Characterization of evolving
patterns
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent

Characterization of local dynamic patterns

Particle collision characterisation
Free particle 2-particle front collision
2-particle side collision 3-particle collision 4-particle side collision

Dynamics of a 4-particle
collision
New particles
Fixed point
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent Patterns and Colliding Particles in the Game
of Life. In A. Adamatzky (ed), From Parallel to Emergent Computin, Taylor & Francis / CRC, 2018

Algorithmic dynamics of a stable
‘near miss’
Zenil et al., Algorithmic-information Properties of Dynamic Persistent

Sensitivity of dynamics to initial conditions
Periodic fixed point
Zenil et al., Algorithmic-information Properties of Dynamic Persistent

Particle annihilation
H. Zenil, N.A. Kiani and J. Tegnér, Algorithmic-
information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of
Life, In A. Adamatzky (ed), From Parallel to
Emergent Computing (book), Taylor & Francis /
CRC Press, 2018

All cases can be reduced to 4 density
plots (+ annihilation)
Algorithmicdynamics
ofglidercollisions
0 50 100 150 200
50
100
500
1000
BDM all cases

Moving
Objects Towards
& Away From
Randomness
Dr. Hector Zenil

A Causal Calculus Based on
Algorithmic Information Dynamics
Let S be a non-random binary file containing a string in binary:
S= 10101010101010101010101010101010101010101010101010101010101010
Clearly, S is algorithmically compressible, with pS = “Print(01) 31 times” a short
computer program generating S. Let S’ be equal to S except for a bitwise operation
(bitwise NOT, or complement) in, say, position 23:
S’ = 10101010101010101010100010101010101010101010101010101010101010
We have that the relationship of their algorithmic complexity denoted by C is:
C(S) < C(S’)
for any single-bit mutation of S, where C(S) = |pS| and C(S’) = |pS’| are the lengths
of the shortest computer programs generating S and S’.

The algorithmic information dynamics of a string (step 0) pushed towards and away
from randomness by digit removal using BDM.
The initial string consists of a simple segment of ten 1s followed by a random-looking
segment of 10 digits. The resulting strings mostly extract each of the respective
segments.
Moving a string

Neutral Deletion
Neutral digit deletion maximises the preservation of the elements contributing to
the algorithmic information content of the original string thus the most important
(computable) features (of which statistical regularities are a subset). Here applied
to a string (step 0) after
removal of 10 digits.

Random data is of
different nature
Now consider S to be a binary file of the same size but consisting of, say,
random data:
01101100100011001101010001110110001100011010011100010000110
So pS = Print(S) is the shortest possible generating program, then, in a
random object:
All perturbations are neutral!
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming
Systems, bioaRXiv DOI: https://doi.org/10.1101/185637

Identifying random objects by
the effect of perturbations
01101100100011001101010001110110001100011010011100010000110
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S.
Elias, A. Schmidt, G. Ball, J. Tegnér, An
Algorithmic Information Calculus for Causal
Discovery and Reprogramming
Systems, bioaRXiv DOI:
https://doi.org/10.1101/185637

The Principle of Algorithmic
Information Dynamics
C(rule 30)
C(rule 30 after 1 step) = C(rule 30 after 2 steps) + log(2) = …
C(rule 30 after 100 steps) + log(100) … = C(rule 30 after 1000 steps) + log(1000)
In a deterministic isolated dynamical system algorithmic complexity never
changes if it did, then it is neither deterministic or isolated and we can
identify those elements

Networks as programs
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, bioaRXiv DOI:
https://doi.org/10.1101/185637

Information Ranking and
Information Spectra
https://doi.org/10.1101/185637

Information Signatures
σ(G)
H Zenil, N.A. Kiani, F. Marabita, Y.
Deng, S. Elias, A. Schmidt, G. Ball,
J. Tegnér,
An Algorithmic Information Calculus
for Causal Discovery and
Reprogramming Systems, bioaRXiv
DOI: https://doi.org/10.1101/185637

https://doi.org/10.1101/185637

Moving Networks
https://doi.org/10.1101/185637

Reconstructing
Dynamical
Systems
Dr. Hector Zenil

AID Application to
Dynamical Systems
Original
candidate
model set M
with individual
models mi in M
After negative
perturbation:
candidate
models M’
grow in length (in
average):
Σi=1
|M’||m’i|>Σi=1
|M||mi|
Positive
perturbation:
candidate
models
decrease in length
Neutral
perturbation
|M| is the
cardinality of M
and |mi| the size
of model m in M.

Reconstructing space-time
dynamics of ECAs
original
reconstructed
https://doi.org/10.1101/185637

Reconstructing time in space-time
dynamics
We are generalizing these tools to continuous dynamical systems such as strange attractors
original
reconstructed
https://doi.org/10.1101/185637

Reconstructing space-phase dynamics of
ECA (more observations)
https://doi.org/10.1101/185637

Sensitivity to perturbations at
difference time steps
https://doi.org/10.1101/185637

System’s
Reprogrammability
Dr. Hector Zenil

Evidence of pervasive
Reprogrammability
Reprogrammability of ECA and
CA
J. Riedel, H. Zenil
Cross-boundary Behavioural Reprogrammability Reveals Evidence of Pervasive Universality
International Journal of Unconventional Computing, vol 13:14-15 pp. 309-357

J. Riedel, H. Zenil
Cross-boundary Behavioural Reprogrammability Reveals Evidence of Pervasive Universality
International Journal of Unconventional Computing, vol 13:14-15 pp. 309-357
Reprogrammability Networks
Evidence of pervasive
Turing universality

J. Riedel and H. Zenil
Rule Primality, Minimal Generating Sets and Turing-Universality in the Causal Decomposition of Elementary Cellular
Automata
Journal of Cellular Automata, vol. 13, pp. 479–497, 2018
New Universal
Cellular Automata

An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér
doi: https://doi.org/10.1101/185637
Algorithmic space Dynamical phase space
Algorithmic/Dynamic
Landscape Relationship

Information
signature of
complete graph
Information
signature of E-R
random graph
K(E-R)
~ |E(G)|K(k)
~ log|V(G)|
Reprogrammability Asymmetry

Reprogrammability Space
Pr(G) x PA(G)

Unit 6: All

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Unit 6: All

Similar to Unit 6: All (20)

More from Hector Zenil

More from Hector Zenil (20)

Recently uploaded

Recently uploaded (20)

Unit 6: All