This is the slides of my master defense; 17 april 2003 subject: "High capacity neural network optimization problems: study & solutions exploration"

1. High capacity neural network optimization problems: study & solutions exploration Francis Piéraut, eng., M.A.Sc [email_address] http://fraka6.blogspot.com/

4. sortie z cible t t 1 t k y 1 x i x D y N w kj w ij x 1 Neural Networks and capacity P(c i |x i ) P(c i |x i ) y 2 y j z 1 Z k

6. High/huge capacity Neural Network y 1 y 2 y 2

8. Errors :Optimization Inefficiency of High Capacity Neural Networks

9. CPU time: Optimization Inefficiency of High Capacity Neural Networks

11. sortie z cible t z 1 Z k t 1 t k y 1 x i x D y N w kj w ij x 1 Neural Networks and equations y 2 y j

12. Learning process is slowing down for non-linear relationships

13. Solutions space of a N+K Neurones Neural Network Solution space of a N Neurones Neural Network Solutions space

14. Similar Solutions Initial State Example 5 iterations 3 iterations

18. Incremental Neural Networks : first approach

19. Incremental Neural Networks : first approach (fix weights optimisation)

20. Hypothesis: Incremental NN OK Incremental NN Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution

21. Incremental Neural Networks (1): results

22. Why it doesn’t work? (critical points)

25. Incremental Neural Network : second approach (add hidden layers) z 1 z 2 x 1 x 2 z 1 z 2 y 1 x 1 x 2 y 2 y 3 y 4

26. Cost function curve shape

27. Hypothesis: Incremental NN (add layers) OK Incremental NN (add layers) Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution

28. Incremental Neural Network (2): results

29. Uncoupled architecture

30. Hypothesis: Uncoupled Architecture OK Removed Decoupled architecture Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution

31. In efficiency of high capacity Neural Networks (CPU time)

32. Efficiency of High capacity Neural Network: decoupled architecture

33. Hypothesis: Partial Parameters optimization OK Opt. partie Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution

34. Neural Networks with partial parameters optimization: results All parameters optimization Max sensitivity optimization

35. Why predicting parameters? (observation) Époque Valeurs

36. Hypothesis * Benefit: reduce # iterations by predicting values based on history Parameter prediction Symetry Gradient dillution Specialisation mechanism Opposite gradient Moving target Problems Solution

37. Prediction : Quadratic extrapolation

38. Prediction : Learning rate increase

42. Any Questions??

43. Exemple :solution linéaire

44. Exemple :solution hautement non-linéaire

45. Sélection des connections influençant le plus le coût

46. Sélection des connections influençant le plus l’erreur T = 1 S = 0 T = 0 S = 1 T = 0 S = 0.1 T = 0 S = 0.1

47. Observation: idealized behavior of the ratio time