The document discusses backpropagation and how it is used to train multilayer perceptron neural networks. Backpropagation is a method used to calculate the gradient of the loss function with respect to the network parameters in order to update the weights in the direction of steepest descent during training. It works by propagating errors backwards from the final layer through the network to earlier layers to compute sensitivity values. The weights are then updated using these sensitivities and the gradient of the loss function to minimize error. An example 1-2-1 network is also described to illustrate forward propagation, backpropagation of errors to compute sensitivities, and weight updates.

1. Backpropagation

2. Multilayer Perceptron R – S 1 – S 2 – S 3 Network

3. Example

4. Elementary Decision Boundaries First Subnetwork First Boundary: Second Boundary:

5. Elementary Decision Boundaries Third Boundary: Fourth Boundary: Second Subnetwork

6. Total Network

7. Function Approximation Example Nominal Parameter Values

8. Nominal Response

9. Parameter Variations

10. Multilayer Network

11. Performance Index Training Set Mean Square Error Vector Case Approximate Mean Square Error (Single Sample) Approximate Steepest Descent

12. Chain Rule Example Application to Gradient Calculation

13. Gradient Calculation Sensitivity Gradient

14. Steepest Descent Next Step: Compute the Sensitivities (Backpropagation) s m F ˆ  n m  - - - - - - - - - -  F ˆ  n 1 m  - - - - - - - - - F ˆ  n 2 m  - - - - - - - - -  F ˆ  n S m m  - - - - - - - - - - - =

15. Jacobian Matrix n m 1 +  n m  - - - - - - - - - - - - - - - - - n 1 m 1 +  n 1 m  - - - - - - - - - - - - - - - - n 1 m 1 +  n 2 m  - - - - - - - - - - - - - - - -  n 1 m 1 +  n S m m  - - - - - - - - - - - - - - - - n 2 m 1 +  n 1 m  - - - - - - - - - - - - - - - - n 2 m 1 +  n 2 m  - - - - - - - - - - - - - - - -  n 2 m 1 +  n S m m  - - - - - - - - - - - - - - - -    n S m 1 + m 1 +  n 1 m  - - - - - - - - - - - - - - - - n S m 1 + m 1 +  n 2 m  - - - - - - - - - - - - - - - -  n S m 1 + m 1 +  n S m m  - - - - - - - - - - - - - - - -  F Ý m n m   f Ý m n 1 m   0  0 0 f Ý m n 2 m    0    0 0  f Ý m n S m m   =

16. Backpropagation (Sensitivities) The sensitivities are computed by starting at the last layer, and then propagating backwards through the network to the first layer.

17. Initialization (Last Layer) a i  n i M  - - - - - - - - - - a i M  n i M  - - - - - - - - - - f M n i M    n i M  - - - - - - - - - - - - - - - - - - - - - - - f Ý M n i M   = = = s i M 2 t i a i –   – f Ý M n i M   =

18. Summary Forward Propagation Backpropagation Weight Update

19. Example: Function Approximation 1-2-1 Network + - t a e p

20. Network 1-2-1 Network a p

21. Initial Conditions

22. Forward Propagation

23. Transfer Function Derivatives

24. Backpropagation

25. Weight Update

26. Choice of Architecture 1-3-1 Network i = 1 i = 2 i = 4 i = 8

27. Choice of Network Architecture 1-5-1 1-2-1 1-3-1 1-4-1

28. Convergence 1 2 3 4 5 0 1 2 3 4 5 0

29. Generalization 1-2-1 1-9-1

30. Dudas ???

31. Hasta la próxima !!!