Types of Activation functions

1. Types of Activation
Functions
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 1

2. Activation Function
• The activation function decides whether a neuron should be activated or not
by calculating the weighted sum and further adding bias to it. The purpose of
the activation function is to introduce non-linearity into the output of a
neuron.
• Types of Non-linear Activation Function 1.Sigmoid 2.tanh 3.ReLU
What’s the “Best”Activation Function?
• Initially: sigmoid was popular
• Then: tanh became popular
• Now: RELU is preferred (better results)
• Softmax: for FC (fully connected) layers
• sigmoid and tanh are used in LSTMs
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 2

3. The sigmoidActivation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 3

4. The tanhActivation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 4

5. The ReLU Activation
Function
The softmax Activation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 5