The document introduces the S-shaped Rectified Linear (SReLU) activation function for deep learning. The SReLU function has four trainable parameters and generalizes the ReLU function by allowing it to be convex or non-convex, emulating human perception mechanisms. The document describes experiments using SReLU with CNNs on CIFAR10 and MNIST datasets, finding SReLU achieves better accuracy than other activation functions.

1. Deep Learning with S-shaped
Rectified Linear Activation
Units
Andrea Lombardo
Luca Tomei
1

2. USED ACTIVATION FUNCTIONS
• RELU: max 0, 𝑥𝑖
• PARAMETRIC RELU: min 0, 𝑎𝑥𝑖 + max 0, 𝑥𝑖
• LEAKY RELU : min 0, 0.01𝑥𝑖 + max 0, 𝑥𝑖
• ELU: 𝑓 𝑥 =
𝑎 𝑒𝑥
− 1 ∀𝑥 ≤ 0
𝑥 ∀𝑥 ≥ 0
• PARAMETRIC ELU: 𝑓 𝑥 =
𝑎 𝑒
𝑥
𝑏 − 1 ∀𝑥 ≤ 0
𝑎
𝑏
𝑥 ∀𝑥 ≥ 0
• SRELU
2

3. INTRODUCTION OF THE SRELU FUNCTION
S-shaped Rectified Linear Unit
3

4. PROPERTIES OF THE SRELU FUNCTION
The SReLU activation function has some remarkable properties:
It has four trainable parameters:
4

5. PROPERTIES OF THE SRELU FUNCTION
The SReLU activation function has some remarkable properties:
It generalizes the ReLU function:
5

6. PROPERTIES OF THE SRELU FUNCTION
The SReLU activation function has some remarkable properties:
It can be settled to be either convex or non-convex:
6

7. PROPERTIES OF THE SRELU FUNCTION
The SReLU activation function has some remarkable properties:
It can be settled to emulate the human perception mechanisms:
7

8. EXPERIMENTAL SETTINGS
8

9. IMPLEMENTED ARCHITECTURES ON CIFAR10
CNN 1 CNN 2
9

10. IMPLEMENTED ARCHITECTURES ON MNIST
CNN 1 CNN 2
10

11. DATASETS
CIFAR10 MNIST
11

12. IMPLEMENTATION DETAILS
12

13. NEURAL NETWORKS IMPLEMENTATION
CNN1 on CIFAR10 CNN1 on MNIST
13

14. NEURAL NETWORKS IMPLEMENTATION
CNN2 on CIFAR10 CNN2 on MNIST
14

15. MySReLU IMPLEMENTATION
15

16. MySReLU IMPLEMENTATION
tl tr
16

17. RESULTS
17

18. RESULTS: CNN1 ON CIFAR10
ACCURACY: rectified vs exponential activation functions
18

19. RESULTS: CNN1 ON CIFAR10
LOSS: rectified vs exponential activation functions
19

20. RESULTS: CNN2 ON MNIST
ACCURACY: rectified vs exponential activation functions
20

21. RESULTS: CNN2 ON MNIST
LOSS: rectified vs exponential activation functions
21

22. RESULTS: CNN2 ON CIFAR10
ACCURACY & LOSS
22

23. RESULTS: CNN1 ON MNIST
ACCURACY & LOSS
23

24. Grazie dell’attenzione
24