In biological systems, individual phenotypes are typically adopted by multiple genotypes. Examples include protein structure phenotypes, where each structure can be adopted by a myriad individual amino acid sequence genotypes. These genotypes form vast connected ‘neutral networks’ in genotype space. The size of such neutral networks endows biological systems not only with robustness to genetic change, but also with the ability to evolve a vast number of novel phenotypes that occur near any one neutral network. Whether technological systems can be designed to have similar properties is poorly understood.
Here we ask this question for a class of programmable electronic circuits that compute digital logic functions. We demonstrate that circuits that compute the same logic function are connected in large neutral networks that span circuit space. Their robustness or fault-tolerance varies very widely. The vicinity of each neutral network contains circuits with a broad range of novel functions. Two circuits computing different functions can usually be converted into one another via few changes in their architecture. These observations show that properties important for the evolvability of biological systems exist in a commercially important class of electronic circuitry. They also point to generic ways to generate fault-tolerant, adaptable and evolvable electronic circuitry, likely to find use in synthetic biology as well.
Robustness and evolution of novel functions in programmable hardware
1. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
.
.
. ..
.
.
Robustness and evolution of novel functions
in programmable hardware
Karthik Raman
Andreas Wagner Lab
Institute of Evolutionary Biology and Environmental Studies
University of Zürich
April 1, 2011
1 / 40
2. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Outline
.
. .1 Introduction
Robustness and Evolvability
Neutral Networks
.
. .2 Genotype Space and Neutral Networks
Genotype space
Phenotype space
Neutral networks
.
. .3 Robustness and Evolvability
Robustness of circuits
Evolving new functions
.
. .4 Summary
2 / 40
3. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Biological systems vs. Man-made systems
.
Biological systems
..
.
. ..
.
.
Shaped through the forces of mutation and natural selection
Survive onslaught of disruptive agents — hostile
environments/random mutations: robust
Show a remarkable ability to adapt and evolve novel properties
through such random mutations: evolvable
.
Man-made systems
..
.
. ..
.
.
Product of rational design, rather than biological evolution
Generally not as robust: often fragile — modification/removal of
components results in catastrophic failure
Their ability to acquire novel and useful features through random
change is limited
3 / 40
4. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Biological systems vs. Man-made systems
.
Biological systems
..
.
. ..
.
.
Shaped through the forces of mutation and natural selection
Survive onslaught of disruptive agents — hostile
environments/random mutations: robust
Show a remarkable ability to adapt and evolve novel properties
through such random mutations: evolvable
.
Man-made systems
..
.
. ..
.
.
Product of rational design, rather than biological evolution
Generally not as robust: often fragile — modification/removal of
components results in catastrophic failure
Their ability to acquire novel and useful features through random
change is limited
3 / 40
5. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Biological systems vs. Man-made systems
.
Biological systems
..
.
. ..
.
.
Shaped through the forces of mutation and natural selection
Survive onslaught of disruptive agents — hostile
environments/random mutations: robust
Show a remarkable ability to adapt and evolve novel properties
through such random mutations: evolvable
.
Man-made systems
..
.
. ..
.
.
Product of rational design, rather than biological evolution
Generally not as robust: often fragile — modification/removal of
components results in catastrophic failure
Their ability to acquire novel and useful features through random
change is limited
3 / 40
6. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Robustness
Fundamental feature of most biological systems — ability to continue
normal function in the face of perturbations
Man-made systems — generally not very robust to failure
Design of fault-tolerant systems is receiving increased attention
4 / 40
7. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Evolvability
“The ability to produce phenotypic diversity, novel solutions to the problems
faced by organisms and evolutionary innovations”
Common feature of biological systems — novel functions, which help an
organism survive and reproduce, are acquired through genetic change.
The need for adaptation in artificial systems is being recognised — ability
to learn and adapt to new situations, e. g. autonomous robots
5 / 40
8. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Evolvable Hardware
Principles of evolution have been applied in computer science for
solving many complex optimisation problems — evolutionary
computationa
implements aspects such as random variation, reproduction,
selection, . . .
Artificial evolution can automatically generate designs of digital
circuits, as well as circuits that are robust to noise and faultsb
Evolvable hardware = evolutionary algorithms applied to electronic
circuits and devices
a
Foster JA (2001) Nat Rev Genet 2:428–436
b
Hartmann M & Haddow P (2004) IEE Proceedings - Computers and Digital Techniques 151:287–294
6 / 40
9. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Key Questions
How ‘robust’ is a typical circuit to changes in the
wiring/configuration?
Do neutral networks exist in this circuit space?
Can circuits with significantly different configuration compute the
same Boolean function?
Does the organisation of neutral networks facilitate the adoption of
novel phenotypes (logic function computations) through small
numbers of gate changes?
7 / 40
10. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
8 / 40
11. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
8 / 40
12. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
8 / 40
13. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
8 / 40
14. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
8 / 40
15. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
8 / 40
16. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks
Genotype space
Nodes: Genotypes
Edges: mutation/simple genetic
change
Phenotypes mapped to genotypes
Largest neutral network
Smaller neutral network
Smaller and more fragmented neutral
set
Disconnected multiple neutral sets
Neutral networks — Genotypes sharing the same phenotype
The genotype space is covered with multiple neutral sets/networks
Neutral networks have implications for robustness/evolvability
8 / 40
17. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Genotype vs. Phenotype
Genotype evolves through mutation and is inherited; selection acts
on phenotype
Genotype–phenotype mapping — required to distinguish neutral
networks
Mapping may not be straightforward always
System Genotype Phenotype
RNA/Proteins Sequence Structure
Regulatory networks Regulatory interactions Gene expression patterns
Circadian oscillators Regulatory interactions Oscillations
Digital circuits Circuit components/wiring Boolean function computed
Protein–protein interactions Network topology ?
Signal transduction Network topology ?
9 / 40
18. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral networks vs. Robustness/Evolvability
Neutral networks can be traversed in small evolutionary steps,
through simple changes/mutations
Larger neutral network ⇒ higher robustness, scope for evolutionary
innovation …
10 / 40
19. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks vs. Robustness
Time
High robustness
Low robustness
Low robustness: more deleterious mutations
High robustness: likely to encounter more novel phenotypes 11 / 40
20. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks vs. Robustness
Time
High robustness
Low robustness
Low robustness: more deleterious mutations
High robustness: likely to encounter more novel phenotypes 11 / 40
21. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral Networks vs. Robustness
Time
High robustness
Low robustness
—Wagner A (2008a) Nat Rev Genet 9:965–974
Low robustness: more deleterious mutations
High robustness: likely to encounter more novel phenotypes 11 / 40
22. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Robustness vs. Evolvability
Robustness and evolvability — both correlate with neutral network
size/connectivity
Robust phenotypes tend to have higher evolvability
Low Robustness
Low Innovation
High Robustness
Low Innovation
High Robustness
High Innovation
—Ciliberti S et al. (2007) Proc Natl Acad Sci U S A 104:13591–13596
Populations evolving on larger neutral networks have greater access
to variation*
12 / 40
23. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Outline
.
. .1 Introduction
Robustness and Evolvability
Neutral Networks
.
. .2 Genotype Space and Neutral Networks
Genotype space
Phenotype space
Neutral networks
.
. .3 Robustness and Evolvability
Robustness of circuits
Evolving new functions
.
. .4 Summary
13 / 40
24. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Field Programmable Gate Arrays (FPGAs)
I
N
P
U
T
S
1
2
3
4
6
7
O
U
T
P
U
T
S
5 8
9
10 13
12
11
OR
A
B
Y
AND
A
B
Y
NAND
A
B
Y
NOR
A
B
Y
XOR
A
B
Y
Y=A+B
Y=A×B
Y=AÅB
___
Y=(A×B)
____
Y=(A+B)
n × m array of logic gates
Configuration can be dynamically reprogrammed: ‘field
programmable’
Maps a Boolean input of nI variables to an output of no variables:
f : {0, 1}nI
→ {0, 1}no
14 / 40
25. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Generic FPGA
inputs
outputs
nI inputs, mapped to no outputs
All nI inputs must be used in column 1
n × m array of logic gates, Lij, each with two inputs
Inter-connections are feed-forward: a gate in column i can take an
input from any of the previous levels
no outputs can come from any of the mn gate outputs
15 / 40
26. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Representation of FPGAs
I
N
P
U
T
S
1
2
3
4
6
7
O
U
T
P
U
T
S
5 8
9
10 13
12
11
4 2 3
L11
1 1 5
L12
3 2 2
L13
2 2 5
L21
1 7 2
L22
5 2 3
L23
10 5 3
L31
3 3 1
L32
6 6 4
L33
11 12 10 13
Outputs
Size of representation = 3mn + no = diameter of circuit space
[Similar to Cartesian Genetic Programming — Miller & Thomson (2000)]
16 / 40
27. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Genotype space: Circuits with various configurations
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
C1 C2
C3C4C5
C6
I
N
P
U
T
S
1
2
4
53
6
O
U
T
P
U
T
S
Every node in genotype space is a circuit (of a particular size)
Neighbours vary by a small elementary change in: (a) Internal wiring
(C2), (b) Logic function of one of the gates (C3), (c) Input mapping
(C5), or (d) Output mapping (C6)
d(C4, C1) = 1 ⇒ not neighbours
17 / 40
28. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Phenotype Space: Boolean functions
Every circuit maps a Boolean input of nI bits to an output of no bits:
f : {0, 1}nI
→ {0, 1}no
For example,
i j k l ⇒ j k l i
Circular [Left] Shift
i j k l ⇒ 0 i j k
[Unsigned] Right Shift
I1 I2 I3 I4 O1 O2 O3 O4
0 0 0 0 0 0 0 0
0 0 0 1 0 0 1 0
0 0 1 0 0 1 0 0
0 0 1 1 0 1 1 0
0 1 0 0 1 0 0 0
0 1 0 1 1 0 1 0
0 1 1 0 1 1 0 0
0 1 1 1 1 1 1 0
1 0 0 0 0 0 0 1
1 0 0 1 0 0 1 1
1 0 1 0 0 1 0 1
1 0 1 1 0 1 1 1
1 1 0 0 1 0 0 1
1 1 0 1 1 0 1 1
1 1 1 0 1 1 0 1
1 1 1 1 1 1 1 1
Circular left shift
truth table
Total Boolean functions = 2entries in truth table
= 264
= 1.84 × 1019
18 / 40
29. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Sampling circuit space
Sample size = 2 × 107
Circuit space sampled uniformly
FPGA size Genotype
space size
Functions observed
in sample
3 × 3 1.08 ×1024
1.59 ×107
4 × 4 8.43 ×1045
1.74 ×107
5 × 5 2.90 ×1076
1.80 ×107
6 × 6 6.05 ×10116
1.83 ×107
Total Boolean functions = 264
= 1.84 × 1019
Small fraction of total Boolean functions observed: ≈ 10−12
Most observed functions are computed only by one circuit in the
sample
Few functions are computed by large number of circuits
19 / 40
30. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Majority of functions are rare
10
0
10
5
10
−8
10
−7
10
−6
10
−5
10
−4
Rank of logic function (decreasing size of circuit sets)
Frequencyoflogicfunction
Some functions appear over a 1000
times
Long tail: Huge number of functions
appear only once
Functions such as right shift/circular
shift do not appear in the sample of
2 × 107
20 / 40
31. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neutral networks in circuit space
Circuits computing the same function can be reached through
function-preserving random walks in circuit space
⇒ Circuit space is spanned by large neutral networks
0 0.2 0.4 0.6 0.8 1
10
0
10
1
10
2
10
3
Maximal circuit distance (D)
Numberoffunctions
4x4 Circuits
10
−7
10
−6
10
−5
10
−4
0.3
0.5
0.7
1
Frequency of the logic functionMaximalcircuitdistance(D)
4x4 Circuits
Spearman’s r = 0.56; p < 10−300
; n = 1000
A function-preserving random walk in circuit space can take us very
far away from the initial circuit
⇒ Neutral networks have very large diameter
21 / 40
32. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Diverse circuits can compute the same function
I
N
P
U
T
S
1
2
3
4
6
7
O
U
T
P
U
T
S
5 8
9
10 13
12
11
4 2 3 1 1 5 3 2 2 2 2 5 1 7 2 5 2 3 10 5 3 3 3 1 6 6 4 11 12 10 13
I
N
P
U
T
S
1
2
3
4
6
7
O
U
T
P
U
T
S
5 8
9
10 13
12
11
2 3 1 4 4 1 1 1 3 5 3 4 7 6 1 7 3 1 1 1 2 1 5 2 5 2 2 13 10 9 11
Both circuits compute
circular shift
Differ in every logic
gate, wiring, input &
output mapping
Thus, the diameters of
neutral networks can be
quite large
22 / 40
33. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Outline
.
. .1 Introduction
Robustness and Evolvability
Neutral Networks
.
. .2 Genotype Space and Neutral Networks
Genotype space
Phenotype space
Neutral networks
.
. .3 Robustness and Evolvability
Robustness of circuits
Evolving new functions
.
. .4 Summary
23 / 40
34. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Robustness to change in configuration
10
−7
10
−6
10
−5
10
−4
0
0.2
0.4
0.6
0.8
1
Frequency of the logic function
Robustnesstoconfigurationchange
4x4 Circuits
Spearman’s r = 0.32; p < 10
−300
0 0.2 0.4 0.6 0.8 1
0
50
100
Circular shift (3x3)
0 0.2 0.4 0.6 0.8 1
0
20
40
Numberofcircuits
Circular shift (4x4)
0 0.2 0.4 0.6 0.8 1
0
20
40
Fraction of neutral neighbors
Circular shift (6x6)
Robustness = fraction of neutral neighbours
Robustness distribution is quite broad
Larger circuits are more robust (regardless of function)
24 / 40
35. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Robustness to gate failure
10
−7
10
−6
10
−5
10
−4
0
0.2
0.4
0.6
0.8
1
Frequency of the logic function
Robustnesstogatefailure
4x4 Circuits
0 0.2 0.4 0.6 0.8 1
0
50
Circular shift (3x3)
0 0.2 0.4 0.6 0.8 1
0
20
40
Numberofcircuits
Circular shift (4x4)
0 0.2 0.4 0.6 0.8 1
0
20
40
Failure robustness
Circular shift (6x6)
Gate failures are common in digital circuits
Types of failures considered: zero output (and inverted output)
Failure robustness = fraction of single gate failures that do not affect
the computed function
Failure robustness distribution is also quite broad
Larger circuits are more robust to gate failure
25 / 40
36. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Evolving new functions
On large neutral networks, genotypes can change substantially
without changing phenotype
Phenotypes in different neighbourhoods of a neutral network can be
quite different
In biological systems, this feature facilitates the exploration of new
phenotypesab
For circuits, this will have implications for the ease with which
evolvable hardware can acquire new functions
a
Ciliberti S et al. (2007) Proc Natl Acad Sci U S A 104:13591–13596
b
Wagner A (2008b) Proc Biol Sci 275:91–100
26 / 40
37. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neighbourhood of evolving circuits
0 500 1000 1500 2000
0
500
1000
1500
2000
2500
3000
Random walk steps
Cumulativenovelfunctions
encounteredinneighborhood
0 1000 2000
0
1000
2000
3000
f = 5.00 ⋅ 10
−8
Cumulativenovelfunctions
encounteredinneighborhood
0 1000 2000
0
1000
2000
3000
f = 5.00 ⋅ 10
−8
0 1000 2000
0
1000
2000
3000
f = 3.00 ⋅ 10
−7
0 1000 2000
0
1000
2000
3000
f = 4.65 ⋅ 10
−6
0 1000 2000
0
1000
2000
3000
f = 6.35 ⋅ 10−6
Random walk steps
Cumulativenovelfunctions
encounteredinneighborhood
0 1000 2000
0
1000
2000
3000
f = 9.40 ⋅ 10−6
Random walk steps
0 1000 2000
0
1000
2000
3000
f = 1.24 ⋅ 10−5
Random walk steps
0 1000 2000
0
1000
2000
3000
f = 5.09 ⋅ 10−5
Random walk steps
As we explore the neutral network (through a function-preserving
random walk), a large number of new phenotypes are encountered
The number of new phenotypes keeps increasing, as we prolong the
random walk
This property holds for a large number of functions
Indicates the high accessibility of innovation in the neighbourhood
27 / 40
38. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neighbourhood of evolving circuits
What is the fraction of new phenotypes encountered in the
neighbourhood of any circuit?
With reference to a starting circuit C0,
u(C0, Ci) = 1 −
|N0 ∩ Ni|
|N0 ∪ Ni|
28 / 40
39. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neighbourhood of evolving circuits
What is the fraction of new phenotypes encountered in the
neighbourhood of any circuit?
With reference to a starting circuit C0,
u(C0, Ci) = 1 −
|N0 ∩ Ni|
|N0 ∪ Ni|
G1
G2
shared
unique
u(G1, G2) = = 5
8
= 0.625
28 / 40
40. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Neighbourhood of evolving circuits
0 50 100 150 200
0
0.2
0.4
0.6
0.8
1
Random walk steps
Fractionofnewfunctions
inneighborhood
0 1000 2000
0
0.5
1
f = 5.00 ⋅ 10
−8
Fractionofnewfunctions
inneighborhood
0 1000 2000
0
0.5
1
f = 5.00 ⋅ 10
−8
0 1000 2000
0
0.5
1
f = 3.00 ⋅ 10
−7
0 1000 2000
0
0.5
1
f = 4.65 ⋅ 10
−6
0 1000 2000
0
0.5
1
f = 6.35 ⋅ 10−6
Random walk steps
Fractionofnewfunctions
inneighborhood
0 1000 2000
0
0.5
1
f = 9.40 ⋅ 10−6
Random walk steps
0 1000 2000
0
0.5
1
f = 1.24 ⋅ 10−5
Random walk steps
0 1000 2000
0
0.5
1
f = 5.09 ⋅ 10−5
Random walk steps
Beyond the distance of ≈ one network diameter, over 80% of the
functions found in the neighbourhood are new
At increasing distances from a circuit, a very large number of new
phenotypes are accessible; this property holds for a large number of
functions
29 / 40
41. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Fraction of new phenotypes at the end of a random walk
10
−7
10
−6
10
−5
10
−4
0
0.2
0.4
0.6
0.8
1
Frequency of the logic function
Fractionofnewfunctionsin
neighborhoodatendofrandomwalk
4x4 Circuits
0 0.2 0.4 0.6 0.8 1
0
10
20
30
40
50
60
Fraction of new functions in neighborhood
at the end of random walk
Numberofcircuits
Circular shift (4x4)
A lot of innovation is accessible in the neighbourhood, once we walk
sufficiently far awaya
from a starting circuit . . .
a
2,000 steps, in this case
30 / 40
42. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Proximity of neutral networks
How far must one travel in circuit space from one neutral set, to find
another neutral set?
31 / 40
43. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Proximity of neutral networks
How far must one travel in circuit space from one neutral set, to find
another neutral set?
0 0.2 0.4 0.6 0.8 1
0
50
100
150
200
Minimum distance between circuits
computing right shift and circular shift
(as fraction of circuit space diameter)
Pairsofcircuits
0 0.2 0.4 0.6 0.8 1
0
200
400
600
800
Minimum distance between neutral networks
(as fraction of circuit space diameter)
Pairsofneutralnetworksd(CS, RS) ≤ 3!
It is possible to discover new functions through a relatively small
number of elementary changes
31 / 40
44. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Similarity with Biological Systems
The system of circuits studied show intriguing similarities with biological
systems:
Existence of large highly connected neutral networks
Large number of diverse genotypes adopt the same phenotype
Robustness to change in configurations
Large number of novel phenotypes accessible in the neighbourhood
of evolving genotypes
Proximity of different neutral networks
Importance of “junk” non-functional components …
32 / 40
45. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Role of ‘non-functional components’
Many circuits have non-functional gates, system parts that are not involved in the
computation a given circuit carries out
In biological systems, many amino acids in a proteinab
, many regulatory interactions
in a gene regulation circuitc
, and many metabolic reactions in a metabolic reaction
networkd
may appear as ‘non-functional’ or ‘dispensable’
Tempting to call such system parts ‘junk’ parts — however, such parts play a crucial
role for evolvability, and it is precisely their ability to vary freely in some
environments that allows biological systems to evolve novel phenotypes
For example, in laboratory evolution experiments, proteins with new function evolve
often through changes that do not affect the protein’s principal functionb
Circuits of a minimal size may have the merit of computing a function in an elegant
and simple way — at the same time, they would be utterly unevolvable
Evolvability comes at the price of high complexity
a
Bloom JD et al. (2005) Proc Natl Acad Sci U S A 102:606–611
b
Aharoni A et al. (2005) Nat Genet 37:73–76
c
Ciliberti S et al. (2007) Proc Natl Acad Sci U S A 104:13591–13596
d
Matias Rodrigues JF & Wagner A (2009) PLoS Comput Biol 5:e1000613
33 / 40
46. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Outline
.
. .1 Introduction
Robustness and Evolvability
Neutral Networks
.
. .2 Genotype Space and Neutral Networks
Genotype space
Phenotype space
Neutral networks
.
. .3 Robustness and Evolvability
Robustness of circuits
Evolving new functions
.
. .4 Summary
34 / 40
47. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Limitations
Many other factors decide the suitability of a particular circuit, e. g.
power consumption, robustness to temperature variations, and
trade-offs between functional flexibility and performance
Fine differences exist between two silicon chips – circuits evolved on
one silicon chip are not guaranteed to work on anothera
!
Differences exist, between software simulation and hardware
evolution
Some properties of our (or any other) study system may depend on
the choice of representation for a circuit’s architecture
Possible major computational issues with very large circuit sizes
Due to the astronomical numbers of circuits and functions, one
needs to resort to sampling to understand circuit space — not
limiting if one is interested in generic properties of this space
a
Thompson A (1995) In: F Morán, A Moreno, JJ Merelo, & P Chacon (eds.), Advances in Artificial Life:
Proc. 3rd Eur. Conf. on Artificial Life (ECAL95), volume 929 of LNAI, pages 640–656. Springer-Verlag
35 / 40
48. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Summary
A random sampling of circuit space reveals that majority of the
functions are computed by very few circuits
Circuits computing the same function form large connected neutral
networks with large diameters
Diverse circuits can compute the same function
Larger circuits tend to be more robust; the distribution of robustness
is broad
for a given function and size, the most robust configuration may be
useful in applications
A number of new functions are accessible in the neighbourhood of
evolving circuits
evolvable circuits may find application in the design of adaptive
devices
Neutral networks are also located close together in space
access to new functions through few changes
‘minimal reconfiguration’ may be particularly useful/efficient
36 / 40
49. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Summary
A random sampling of circuit space reveals that majority of the
functions are computed by very few circuits
Circuits computing the same function form large connected neutral
networks with large diameters
Diverse circuits can compute the same function
Larger circuits tend to be more robust; the distribution of robustness
is broad
for a given function and size, the most robust configuration may be
useful in applications
A number of new functions are accessible in the neighbourhood of
evolving circuits
evolvable circuits may find application in the design of adaptive
devices
Neutral networks are also located close together in space
access to new functions through few changes
‘minimal reconfiguration’ may be particularly useful/efficient
36 / 40
50. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Summary
A random sampling of circuit space reveals that majority of the
functions are computed by very few circuits
Circuits computing the same function form large connected neutral
networks with large diameters
Diverse circuits can compute the same function
Larger circuits tend to be more robust; the distribution of robustness
is broad
for a given function and size, the most robust configuration may be
useful in applications
A number of new functions are accessible in the neighbourhood of
evolving circuits
evolvable circuits may find application in the design of adaptive
devices
Neutral networks are also located close together in space
access to new functions through few changes
‘minimal reconfiguration’ may be particularly useful/efficient
36 / 40
51. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Summary
A random sampling of circuit space reveals that majority of the
functions are computed by very few circuits
Circuits computing the same function form large connected neutral
networks with large diameters
Diverse circuits can compute the same function
Larger circuits tend to be more robust; the distribution of robustness
is broad
for a given function and size, the most robust configuration may be
useful in applications
A number of new functions are accessible in the neighbourhood of
evolving circuits
evolvable circuits may find application in the design of adaptive
devices
Neutral networks are also located close together in space
access to new functions through few changes
‘minimal reconfiguration’ may be particularly useful/efficient
36 / 40
52. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Summary
A random sampling of circuit space reveals that majority of the
functions are computed by very few circuits
Circuits computing the same function form large connected neutral
networks with large diameters
Diverse circuits can compute the same function
Larger circuits tend to be more robust; the distribution of robustness
is broad
for a given function and size, the most robust configuration may be
useful in applications
A number of new functions are accessible in the neighbourhood of
evolving circuits
evolvable circuits may find application in the design of adaptive
devices
Neutral networks are also located close together in space
access to new functions through few changes
‘minimal reconfiguration’ may be particularly useful/efficient
36 / 40
53. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Outlook: Exploiting robustness and evolvability
.
In technological systems
..
.
. ..
.
.
For any desired Boolean function, it may be beneficial to use the
most robust circuit (one that is resistant to a number of gate failures
as well as configuration changes) of a particular size
Circuits that have access to a greater amount of innovation (diverse
Boolean functions), may be useful in designing systems with adaptive
behaviour
.
In biology …
..
.
. ..
.
.
Such circuits can also be viewed as generalised models of signalling
networks/gene regulatory networks
Such circuits also find applications in synthetic biology, for the
design of synthetic circuitsa
a
Marchisio MA & Stelling J (2011) PLoS Comput Biol 7:e1001083+
37 / 40
54. . . . . . . . . . .
Introduction
. . . . . . . . .
Genotype Space and Neutral Networks
. . . . . . . . . .
Robustness and Evolvability
. . . . . .
Summary
Not all circuits computing the same function are equal …
I
N
P
U
T
S
1
2
3
4
6
7
O
U
T
P
U
T
S
5
8
Circuit A
Robustness = 0.0556
I
N
P
U
T
S
1
2
3
4
O
U
T
P
U
T
S
1915117
2016128
1814106
171395
Circuit B
Robustness = 0.5725
…is complexity the price for evolvability?
38 / 40