I summarized work in the Friedlander et al. Evolution of Bow-Tie Architectures in Biology (2015) for those who do not have a background in system biology.

1. July/7/2021 Journal club
Thoma Itoh
1

2. Network having bow-tie structure
2

3. Previous work by authors
Authors suggested evolutionary mechanism of modular network
Modular network
Evolutionary pressure
toward modular network
Mutations tend to
eliminate connections
+
Friedlander et al., 2013
https://commons.wikimedia.org/wiki/File:Network_Community_Structure.png
Modular network
3

4. Question inspired by previous study
Whether one can find situations in which evolution leads to bow-tie architectures
Bow-tie architecture
?
Bow-tie architecture
4

5. Outline
Matrix expression of network
Evolutionary simulation; linear network
Robustness of simulation result to the fluctuation
Evolutionary simulation; non-linear network
Discussion
5
?

6. Outline
Matrix expression of network
6
?

7. 𝑨𝒊𝒋
(𝒍)
Input from node j in layer L to node i in layer L + 1
Matrix expression of linear network model
𝑨𝟏𝟏
(𝟏)
𝑨𝟏𝟐
(𝟏)
𝑨𝟏𝟑
(𝟏)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟏
(𝟏)
𝑨𝟑𝟐
(𝟏)
𝑨𝟑𝟑
(𝟏)
Layer 1
Layer 2
Row i: Input vector of node i
7

8. 𝑨𝒊𝒋
(𝒍)
Input from node j in layer L to node i in layer L + 1
Row i: Input vector of node i
Matrix expression of linear network model
𝑨𝟏𝟏
(𝟏)
𝑨𝟏𝟐
(𝟏)
𝑨𝟏𝟑
(𝟏)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟏
(𝟏)
𝑨𝟑𝟐
(𝟏)
𝑨𝟑𝟑
(𝟏)
Layer 1
Layer 2
𝑨𝟏𝟏
(𝟏)
𝑨𝟏𝟐
(𝟏)
𝑨𝟏𝟑
(𝟏)
8

9. 𝑨𝒊𝒋
(𝒍)
Input from node j in layer L to node i in layer L + 1
Matrix expression of linear network model
𝑨𝟏𝟏
(𝟏)
𝑨𝟏𝟐
(𝟏)
𝑨𝟏𝟑
(𝟏)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟏
(𝟏)
𝑨𝟑𝟐
(𝟏)
𝑨𝟑𝟑
(𝟏)
Layer 1
Layer 2
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
Row i: Input vector of node i
9

10. 𝑨𝒊𝒋
(𝒍)
Input from node j in layer L to node i in layer L + 1
Matrix expression of linear network model
𝑨𝟏𝟏
(𝟏)
𝑨𝟏𝟐
(𝟏)
𝑨𝟏𝟑
(𝟏)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟏
(𝟏)
𝑨𝟑𝟐
(𝟏)
𝑨𝟑𝟑
(𝟏)
Layer 1
Layer 2
𝑨𝟑𝟏
(𝟏)
𝑨𝟑𝟐
(𝟏)
𝑨𝟑𝟑
(𝟏)
Row i: Input vector of node i
10

11. 𝑨𝒊𝒋
(𝒍)
Input from node j in layer L to node i in layer L + 1
Matrix expression of linear network model
Row i: Input vector of node i
11

12. Total input-output relationship is product of each layer matrix
=
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2 𝐴 1 = Total input/output
𝐴 2
𝐴 1
𝐴 2
𝐴 1
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟐)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟏)
Output 1
Output 2
Output 3
Output 1 Output 2 Output 3 12

13. Total input-output relationship is product of each layer matrix
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2 𝐴 1 = Total input/output
=
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2
𝐴 1
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
Output 1
Output 2
Output 3
Output 1 Output 2 Output 3 13

14. Total input-output relationship is product of each layer matrix
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
=
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2 𝐴 1 = Total input/output
𝐴 2
𝐴 1
𝐴 2
𝐴 1
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
Output 1
Output 2
Output 3
Output 1 Output 2 Output 3 14

15. Total input-output relationship is product of each layer matrix
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏) 𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
=
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2 𝐴 1 = Total input/output
𝐴 2
𝐴 1
𝐴 2
𝐴 1
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟑
(𝟏)
Output 1
Output 2
Output 3
Output 1 Output 2 Output 3 15

16. Total input-output relationship is product of each layer matrix
x
𝑨𝟑𝟐
(𝟐)
𝑨𝟏𝟐
(𝟐)
x
𝑨𝟐𝟐
(𝟐)
𝑨𝟏𝟐
(𝟐)
𝐴 2 𝐴 1 = Total input/output
𝐴 2
𝐴 1
→ Rank: 1
Rank:
Number of independent rows
=
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟐𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟑
(𝟏)
𝐴 2
𝐴 1
𝑨𝟏𝟐
(𝟐)
𝑨𝟐𝟐
(𝟐)
𝑨𝟑𝟐
(𝟐)
𝑨𝟐𝟏
(𝟏)
𝑨𝟐𝟐
(𝟏)
𝑨𝟑𝟐
(𝟏)
Output 1 Output 2 Output 3 16
Dimension: 3
Rank: 1
Rank deficient: 2

17. Outline
Evolutionary simulation; linear network
17
?

18. Evolutionary simulation
Goal matrix
Fitness
Mutation
Simulaton flow
18

19. Goal matrix is task that network should perform
Task performed by bow-tie architecture
This time, task is given at first
or or
Goal matrix (G)
𝐴11 𝐴12 𝐴23
𝐴21 𝐴22 𝐴23
𝐴31 𝐴32 𝐴33
𝐴11 𝐴12 𝐴23
𝐴21 𝐴22 𝐴23
𝐴31 𝐴32 𝐴33
etc...
Networks that realizes the given task.
19

20. Goal matrix rank reflects the property of given task
3 0 0
0 4 0
0 0 2
2 2 2
4 4 4
6 6 6
Rank 3 goal matrix:
3 independent output
20
Input 1 Input 2 Input 3
Output A Output B Output C
Rank1 goal matrix:
3 dependent output
All outputs depend on different inputs All outputs depend on same inputs
Input 1 Input 2 Input 3
Output A Output B Output C

21. Fitness is distance between total in/out and given desired goal
Evolve network so that realizes the given task ( goal matrix (G) )
Fitness: 𝐹 = − 𝐴(𝐿)𝐴(𝐿−1) … 𝐴(1) − 𝐺
Total in-out: 𝐴(𝐿)
𝐴(𝐿−1)
… 𝐴(1)
Goal matrix: G
Evolve so that
close to the G 10 10 10
10 10 10
10 10 10
20 5 5
20 5 15
5 5 20
21

22. Mutation is biased to eliminate connections
𝐴𝑖𝑗 → 𝐴𝑖𝑗 x N(1, 𝜎)
log(𝐴𝑖𝑗) → log(𝐴𝑖𝑗) + log[ N(1, 𝜎) ]
Product-rule mutation Cumulative distirubtion function
log[ N(1, 𝜎) ]
Biological mutations are likely to eliminate connections
Fundamental reason of product-rule mutation:
Affinity and reaction rate are exponential in free energy ( ∝ Mutation level)
𝜎: 0.01 − 1
https://www.jstage.jst.go.jp/article/mssj/17/2/17_92/_pdf/-char/ja
22

23. ・
・
・
・
・
・
・
・
・
Mutated elements
are rondomly picked
Mutation
・
・
・
2N
P = 0.2
Evolutionary simulation flow
・
・
・
・
・
・
2N
N
Figures are cited and modified from Friedlander et al., 2013
23

24. Bow tie evolves when the goal has deficient rank
Goal matrix
Rank
Smallest width layer is on middle layer
Fewer nodes 6 node
Possible network structure under given goal matrix
Deficient rank
24

25. Bow tie evolves when the mutation is biased to eliminate connections
𝐴𝑖𝑗 → 𝐴𝑖𝑗 x N(1, 𝜎)
Product- rule mutation: Eliminate interaction Sum-rule mutation: Maintain interaction
𝐴𝑖𝑗 → 𝐴𝑖𝑗 + N(0, 𝜎)
Fewer nodes 6 node
25

26. Bow-tie evolution is robust to the evolutionary parameter
Bow-ties were obtained in all cases.
(Tournament size)
Bow tie evolves when...
- The goal has deficient rank
Mutation is biased to eliminate connections
26

27. Outline
Robustness of simulation result to the fluctuation
27
?

28. Test the robustness of evolutionary mechanism to the fluctuation
Suggested evolutionary mechanism
Bow tie evolves when...
- The goal has deficient rank
Mutation is biased to eliminate connections
Test the robustness of this results to the fluctuation
Effect of rank accuracy of goal matrix
Effect of temporal fluctuation over network and goal
28

29. Evolutionary simulation is robust to rank accuracy
Same result with non-noise simulation
( Up to noise strength 1%)
clean
10 10 10 10 10 10
10 10 10 10 10 10
10 10 10 10 10 10
10 10 10 10 10 10
10 10 10 10 10 10
10 10 10 10 10 10
 
 
 
 
  
 
 
 
 
G
noisy
9.9590 10.1611 10.0919 10.0580 9.8794 10.1750
9.9630 9.9266 10.0948 10.0780 10.0377 9.8305
9.9102 9.9060 10.0166 10.0859 9.8447 10.0698
9.94

G
37 10.0309 9.9075 9.8582 9.8845 9.9820
9.8180 10.1331 9.8766 10.0667 9.8966 10.0801
10.0806 10.0267 10.0364 10.0566 9.9908 10.2014
 
 
 
 
 
 
 
 
 
.
Perturbate with noise
Rank deficient
Almost rank deficient
Perturbate goal matrix with noise
29

30. ( ) ( 1) (1)
1 1
||( )( ) ( ) ( ) ||
L L
L L G
F 

      
A ε A ε A ε G ε
𝜀𝑖 ~ 𝑁(0, 𝜎)
10
-4
1
1.5
2
2.5
3
3.5
4
4.5
5
absolute noise added
Mid-layer
width
10
0
1
1.5
2
2.5
3
3.5
4
4.5
5
temporal std in fitness
Mid-layer
width
Add noise to the all matrix entries
Product-mutations filter out the noise much more efficiently
× Sum rule mutation: Maintain interactioins
□ Product rule mutation: Minimize interaction
(std)
Product-mutations filter out the
noise much more efficiently than
sum-mutations.
30

31. Outline
Evolutionary simulation; non-linear network
31
?

32. Bow-ties can evolve in nonlinear information transmission models
4-pixel retina
Proble definition
- Outputs detect whether there is
(i) at least one pixel in the left column is black (Left)
(ii) at least one pixel in the right column is black (Right)
(iii) (i) AND (ii) (Left and Right)
(iv) (i) OR (ii) (Left or Right)
- 4 input
- Two internal processing layer
- Each node performs nonlinear transformation
u(l+1) = f (A(l) u(l) – T(l+1)) u(l) Input
A(l) Weight matrix
T(l+1) Thresholds
Typical results
f(x) = (1 + tanh(x)) / 2 = {0,1
Investigate whether the suggested mechanism would
apply in a nonlinear network model.
32

33. Bow-tie architecture evolves
when realizing rank deficient task
with minimum interaction
Result and discussion
33
Signal1
Gene1
TF
Signal2 Signal3
Gene2 Gene3
Signal1
TF1 TF2 TF3
Signal2 Signal3
Gene1 Gene2 Gene3
Economial Wasteful
Bow-tie architecture