Presentation on a paper which was published in the British International Conference on Databases in 2015. Experimental framework for selecting deep learning hyper-parameters.

1. A Framework for Selecting
Hyper-Parameters
Jim O’ Donoghue
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979
British International Conference on Databases
7th July 2015

2. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979
Background + Motivation
Algorithms + The CDN
Experiments + Results
Future Work
Conclusions
NEED TO FIX NUMBERS

3. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 3
1

4. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 4
1

5. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 5
1
Dementia Awareness
+ Prevention

6. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 6
1
Dementia Awareness + Prevention
Online Environment

7. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 7
1
Dementia Awareness + Prevention
Online Environment
Risk Prediction Algorithm

8. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 8
1
Dementia Awareness + Prevention
Online Environment
Risk Prediction Algorithm
- Validation

9. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 9
3

10. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 10
3

11. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 11
3
High-Dimensional

12. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 12
3
High-Dimensional
Variable
Interactions

13. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 13
3
High-Dimensional
Variable
Interactions

14. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 14
3
High-Dimensional
Variable
Interactions
Hyper-Parameter
Selection

15. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 15
3
High-Dimensional
Variable
Interactions
Hyper-Parameter
Selection

16. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 16
4

17. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 17
4
ClassClass
Input
Features

18. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 18
4
ClassClass
Input
Features
Learned
Features

19. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 19
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

20. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 20
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

21. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 21
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

22. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 22
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

23. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 23
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

24. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 24
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

25. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 25
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

26. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 26
4
Class
Connection
Weights
Class
Input
Features
Learned
Features

27. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 27
4
Connection
Weights
Class
Input
Features
Learned
Features

28. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 28
5
Class
Connection
Weights
Class
Input
Features

29. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 29
6
Connection
Weights
Class
Input
Features

30. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 30
7
Connection
Weights
Input
Features
Learned
Features

31. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 31
8
Class
Connection
Weights
Input
Features
Learned
Features

32. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 32
9

33. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 33
9
MySql
File
System
Configurable Deep Network
Framework

34. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 34
9
MySql
File
System
Grid
Algorithm
Configurable Deep Network
Framework

35. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 35
9
MySql
File
System
Grid
Algorithm
Configurable Deep Network
Framework
e1, e2, … , en

36. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 36
9
MySql
File
System
Grid
Algorithm
Configurable Deep Network
Framework
Query
e1, e2, … , en

37. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 37
9
MySql
File
System
Grid
Algorithm
Configurable Deep Network
Framework
Final Model
Query
e1, e2, … , en

38. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 38
10

39. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 39
11
CDN

40. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 40
12
Subset of the Data – dimensions
What the variables are
What the predictor is
Purpose

41. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 41
13
To Choose:

42. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 42
13
To Choose:
learning rate α

43. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 43
13
To Choose:
learning rate α
weight decay term λ

44. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 44
13
To Choose:
learning rate α
weight decay term λ
training iterations t

45. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 45
13
The Grid:

46. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 46
13
The Grid:
α, λ:
[0.001, 0.003, 0.009, … , 0.1, 0.3, 0.9]

47. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 47
13
The Grid:
α, λ:
[0.001, 0.003, 0.009, … , 0.1, 0.3, 0.9]
t:
[100, 1000, 10000]

48. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 48
13
Categorical Continuous

49. Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 49
13

50. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 50
13
Alpha
0.9 0.3
0.09 0.003
Lambda 0.009 0.003 0.001

51. Lambda 0.009 0.003 0.001
0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 51
13
100
1,000
10,000
Training Iterations
Alpha
0.9 0.3
0.09 0.003

52. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
Categorical Continuous Lambda 0.009 0.003 0.001
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 52
13
100
1,000
10,000
Training Iterations
Alpha
0.9 0.3
0.09 0.003

53. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 53
13
100
1,000
10,000
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

54. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 54
13
100
1,000
10,000
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

55. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 55
13
100
1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

56. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 56
13
100
1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

57. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 57
13
100
1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

58. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 58
13
100
1,000
10,000
0.3046 0.2815
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

59. 0
5
10
15
20
25
30
35
40
45
50
Valid.
Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 59
13
100
1,000
10,000
0.3046 0.2815
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha
0.9 0.3
0.09 0.003

60. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 60
15
To Choose:

61. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 61
15
To Choose:
layer 1 nodes h(1)
n

62. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 62
15
To Choose:
layer 1 nodes h(1)
n
pre-training epochs e

63. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 63
15
The Grid:
h(1)
n:
[10, 30, 337, 900, 1300, 2000]

64. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 64
15
The Grid:
h(1)
n:
[10, 30, 337, 900, 1300, 2000]
e
[1, 5, 10, 15, 20]

65. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 65
15
Parameter Initialisation:
− 4
6
𝑓𝑎𝑛_𝑖𝑛 + 𝑓𝑎𝑛_𝑜𝑢𝑡
, + 4
6
𝑓𝑎𝑛_𝑖𝑛 + 𝑓𝑎𝑛_𝑜𝑢𝑡

66. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 66
15
Epochs 1 5

67. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 67
15
Nodes
10 30 100
337 900 1 300
2000

68. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 68
15
Epochs 1 5
Nodes
10 30 100
337 900 1300
2000

69. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 69
15
Epochs 1 5
Nodes
10 30 100
337 900 1300
2000

70. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 70
15
Epochs 1 5
Nodes
10 30 100
337 900 1300
2000

71. Valid.
Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 71
15
Epochs 1 5
Nodes
10 30 100
337 900 1300
2000
-19.580

72. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 72
14
To Choose:
Last layer nodes h(1)
n

73. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 73
14
To Choose:
Last layer nodes h(1)
n
The Grid:
[10, 30, 337, 900, 1300, 2000]

74. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 74
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous

75. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 75
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Nodes
10 30 100
337 900 1300
2000

76. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 76
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000

77. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 77
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000

78. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 78
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000

79. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 79
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000

80. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 80
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000
0.232

81. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 81
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1300
2000
0.232

82. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 82
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1 1300
2000
0.232

83. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 83
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1 1300
2000
0.232

84. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 84
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1 1300
2000
0.232 0.291

85. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 85
14
0
10
20
30
40
50
60
70
80
90
Valid.
Cost
Categorical Continuous
Nodes
10 30 100
337 900 1 1300
2000
0.232 0.291

86. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 86
16
Lambda @ 0.03
10
2000
200
337
10
200
3567
10
200
3567
10
200
3567
10
200
3567
10
30
10
337
10
100
337
0
1
2
3
4
5
6
7
Alpha 0.001 0.01 0.9

87. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 87
16
Lambda @ 0.03
10
2000
200
337
10
200
3567
10
200
3567
10
200
3567
10
200
3567
10
30
10
337
10
100
337
0
1
2
3
4
5
6
7
Step 3000 1000 100

88. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 88
16
Lambda @ 0.03
10
2000
200
337
10
200
3567
10
200
3567
10
200
3567
10
200
3567
10
30
10
337
10
100
337
0
1
2
3
4
5
6
7
0.265 0.245
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0.272

89. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 89
16
Lambda @ 0.03
10
2000
200
337
10
200
3567
10
200
3567
10
200
3567
10
200
3567
10
30
10
337
10
100
337
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7

90. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 90
18
Activation functions
Algorithms
Inference
Framework – to Mongo and input from
Visualising learning
Implementing Early Stopping
Mini-batch Stochastic Gradient Descent

91. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 91
19
Much easier to model when you have
one extensible network that can handle
many type of data
Constituent models can be used to select
a starting point for deep learning
configurations

92. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 92

93. In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 93
16
Lambda @ 0.03
10
2000
200
337
10
200
3567
10
200
3567
10
200
3567
10
200
3567
10
30
10
337
10
100
337
0
1
2
3
4
5
6
7
0.265 0.245
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0.272