Back propagation using sigmoid & ReLU function

- The document discusses activation functions like sigmoid and ReLU that are used in neural networks. - Sigmoid activation functions have derivatives between 0-0.25, causing the vanishing gradient problem during backpropagation. ReLU functions have derivatives of 0 or 1, avoiding this issue but allowing "dead neurons". - Leaky ReLU was developed to fix the "dead neuron" problem of ReLU by having a small negative slope like 0.01 instead of 0 when the input is negative. This ensures the derivative is never exactly 0.