This document discusses neural network activation distribution and three algorithms (A, B, C) for modeling the process. It analyzes the common behavioral characteristics and provides visual representations. The core characteristics of related software discussed are accuracy, efficiency, and platform independency. Visual results are shown for the algorithms under different parameter settings on Erdos-Renyi graphs.

1. Martin Ayvazyan
Activation distribution in
a neural network
20 August 2018

2. Restricted © 2017 Mentor Graphics Corporation
Contents
 Why are neural networks so important
 Basic network characteristics
 Basic description of the process, formulation of algorithms,
comparison of behavioral characteristics
 Common behavioral characteristics of the process
 A visual representation of the process
 Core characteristics of the software
Activation distribution in a neural network, 20182

3. Restricted © 2017 Mentor Graphics Corporation
Cell Division and Cancer
Activation distribution in a neural network, 20183

4. Restricted © 2017 Mentor Graphics Corporation
The global economic crisis prevention /
elimination
Activation distribution in a neural network, 20184

5. Restricted © 2017 Mentor Graphics Corporation
Traffic control / jams prevention
Activation distribution in a neural network, 20185

6. Restricted © 2017 Mentor Graphics Corporation
Deep learning
Activation distribution in a neural network, 20186

7. Restricted © 2017 Mentor Graphics Corporation
Astronomy
Activation distribution in a neural network, 20187

8. Restricted © 2017 Mentor Graphics Corporation
Basic definitions
ER model
G(N, p)
N – nodes count in the graph
p – component of connectivity
P(G) – the probability that the graph is
connected
Activation distribution in a neural network, 20188
• p =
𝑐∙𝑙𝑛𝑁
𝑁
&& c>1 => graph is connected (c < 1 => graph is disconnected)
• p =
𝑐∙𝑙𝑛𝑁
𝑁
&& (c>3 && N > 100) => 𝑃 𝑁,𝑝 𝐺 ≥ 1 −
1
𝑁

𝑛
𝑁∗ 𝑁−1
2
 𝑃(𝐺0) = 𝑝 𝑛
∙ (1 − 𝑝)
𝑁∙(𝑁−1)
2
−𝑛
 𝑃(𝐺′) =
𝑛
𝑁∙(𝑁−1)
2
∙ 𝑝 𝑛 ∙ (1 − 𝑝)(
𝑁∙ 𝑁−1
2
− 𝑛)

9. Restricted © 2017 Mentor Graphics Corporation
Small world
Activation distribution in a neural network, 20189

10. Restricted © 2017 Mentor Graphics Corporation
Clusterization
Activation distribution in a neural network, 201810
𝐶𝑗 𝑞 𝑗 =
𝑡𝑗
𝑞 𝑗 ∙ (𝑞 𝑗 − 1)/2
𝐶 =
1
𝑁
𝑗
𝐶𝑗

11. Restricted © 2017 Mentor Graphics Corporation
Degree distribution
Activation distribution in a neural network, 201811

12. Restricted © 2017 Mentor Graphics Corporation
Algorithm A
 Input: a network, list of active nodes in the network, μ, λ
 Step 1: If there is no an active node in the network – exit
 Step 2: Randomly is selected a node
 Step 3: If the selected node doesn’t active – go to Step 2
 Step 4: Randomly is selected an adjacent node and in case if the
neighbor doesn’t active – activate it with
𝛌
(𝛌+ 𝛍)
probability
 Step 5: The temporary selected node is deactivated with
𝜇
(𝜆+ 𝜇)
probability
 Step 6: Go to Step 1
Activation distribution in a neural network, 201812

13. Restricted © 2017 Mentor Graphics Corporation
Algorithm A
𝑁=1024, 𝑝=0.5, μ=0.0
Activation distribution in a neural network, 201813

14. Restricted © 2017 Mentor Graphics Corporation
Algorithm A
Activation distribution in a neural network, 201814
P={0.5, 0.1, 0.05} λ = 0.005 P={0.5, 0.1, 0.05} λ = 0.05

15. Restricted © 2017 Mentor Graphics Corporation
Algorithm B
 Input: a network, list of active nodes in the network, μ, λ
 Step 1: If there is no an active node in the network – exit
 Step 2: Randomly is selected a node
 Step 3: If the selected node doesn’t active – go to Step 2
 Step 4: Each adjacent inactive node is activated with
𝛌
(𝛌+ 𝛍)
probability
 Step 5: The temporary selected node is deactivated with
𝜇
(𝜆+ 𝜇)
probability
 Step 6: Go to Step 1
Activation distribution in a neural network, 201815

16. Restricted © 2017 Mentor Graphics Corporation
Algorithm B
Activation distribution in a neural network, 201816
N=1024, p=0.5, μ=0.0

17. Restricted © 2017 Mentor Graphics Corporation
Algorithm B
Activation distribution in a neural network, 201817
N=1024, P = {0.5, 0.1, 0.05}, λ = 0.05 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005

18. Restricted © 2017 Mentor Graphics Corporation
Algorithm C
 Input: a network, list of active nodes in the network, μ, λ
 Step 1: If there is no an active node in the network – exit
 Step 2: For each active note of the current stage:
— Step 2.1: Randomly is selected an adjacent node and in case if the
neighbor doesn’t active – activate it with
𝛌
(𝛌+ 𝛍)
probability
— Step 2.2: The node is deactivated with
𝜇
(𝜆+ 𝜇)
probability
 Step 3: Go to Step 1
Activation distribution in a neural network, 201818

19. Restricted © 2017 Mentor Graphics Corporation
Algorithm C
Activation distribution in a neural network, 201819
N=1024, p=0.5, μ=0.0

20. Restricted © 2017 Mentor Graphics Corporation
Algorithm C
Activation distribution in a neural network, 201820
N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005

21. Restricted © 2017 Mentor Graphics Corporation
Common behavioral characteristics of the process
ER(n,p), active_nodes_count = 1
 The probability that in the current step will be activated at least one
node:
Activation distribution in a neural network, 201821
‘A’ ‘B’ ‘C’
1
𝑛
∙ 𝜆
1
𝑛
∙ (1 − 1 − 𝜆 𝑝∙ 𝑛−1 )
𝜆

22. Restricted © 2017 Mentor Graphics Corporation
Common behavioral characteristics of the process
 The probability that in the current step will be activated m / s.t {1 < m <= p ∙ (n − 1)}
nodes:
 The probability that after the current step in the network there will be no active nodes:
Activation distribution in a neural network, 201822
‘A’ ‘B’ ‘C’
0 1
𝑛
∙ (
𝑚
𝑝 ∙ 𝑛 − 1
∙ 𝜆 𝑚
∙ 1 − 𝜆 𝑝∙ 𝑛−1 −𝑚
) 0
‘A’ ‘B’ ‘C’
μ∙(
1
𝑛
∙ (1 − 𝜆)) μ∙(
1
𝑛
∙ 1 − 𝜆 𝑝∙(𝑛−1)) μ ∙ 1 − 𝜆

23. Restricted © 2017 Mentor Graphics Corporation
A visual representation of the process
ER(n, p) / n=65, p = 005, active_nodes_count=65, 𝜆 = 0.314159265, μ=1.0
Activation distribution in a neural network, 201823
0 20 30
50 500 1500

24. Restricted © 2017 Mentor Graphics Corporation
Core characteristics of the software
 Accuracy
 Efficiency
 Platform independency
Activation distribution in a neural network, 201824

25. Restricted © 2017 Mentor Graphics Corporation
www.mentor.com