The document discusses different types of activation functions used in neural networks. It describes the sigmoid, tanh, and ReLU activation functions as common examples of non-linear functions. While sigmoid was initially popular, ReLU is now preferred as it works better in practice and helps address the vanishing gradient problem. Softmax is used for fully connected layers, while sigmoid and tanh are still used in LSTM networks.

1. Types of Activation
Functions
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 1

2. Activation Function
• The activation function decides whether a neuron should be activated or not
by calculating the weighted sum and further adding bias to it. The purpose of
the activation function is to introduce non-linearity into the output of a
neuron.
• Types of Non-linear Activation Function 1.Sigmoid 2.tanh 3.ReLU
What’s the “Best”Activation Function?
• Initially: sigmoid was popular
• Then: tanh became popular
• Now: RELU is preferred (better results)
• Softmax: for FC (fully connected) layers
• sigmoid and tanh are used in LSTMs
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 2

3. The sigmoidActivation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 3

4. The tanhActivation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 4

5. The ReLU Activation
Function
The softmax Activation Function
3/7/2024 Department of Biomedical Engineering, SRMIST, KTR 5