Downloaded 107 times
![Try it out!
From hands-on session: https://www.dropbox.com/s/92sckhnf1hjgjlo/CNN.zip?dl=0
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
…….
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
Optimizer: SGD, RMSprop, Adagrad, Adam…. (https://keras.io/optimizers/)](https://image.slidesharecdn.com/dloptimization-170715221703/75/Optimization-in-Deep-Learning-28-2048.jpg)
Uploaded byYan Xu
Optimization in Deep Learning
This document summarizes various optimization techniques for deep learning models, including gradient descent, stochastic gradient descent, and variants like momentum, Nesterov's accelerated gradient, AdaGrad, RMSProp, and Adam. It provides an overview of how each technique works and comparisons of their performance on image classification tasks using MNIST and CIFAR-10 datasets. The document concludes by encouraging attendees to try out the different optimization methods in Keras and provides resources for further deep learning topics.
In this document
Powered by AI
Introduction to deep learning and its optimization focus, covering upcoming topics including AlphaGo and attention networks.
Overview of gradient descent and its variants (SGD, mini-batches), performance implications, and recommendations for implementation.
Detailed discussion on momentum, Nesterov's Accelerated Gradient (NAG), AdaGrad, RMSProp, and Adam optimization methods.
Comparative analysis of various optimization techniques using MINIST and CIFAR-10 datasets.
Key learning methods for deep learning and hands-on examples using Keras for model building and optimization.
Summarizes techniques like GD, SGD, and their variants, acknowledging adaptive learning strategies and upcoming sessions.
More Related Content
Similar to Optimization in Deep Learning
Optimization in Deep Learning
- 1. Optimization in DeepLearning Houston Machine Learning Deep Learning Series
- 2. How to getto the lake
- 3. Roadmap • Tour ofmachine learning algorithms (1 session) • Feature engineering (1 session) • Feature selection - Yan • Supervised learning (4 sessions) • Regression models -Yan • SVM and kernel SVM - Yan • Tree-based models - Dario • Bayesian method - Xiaoyang • Ensemble models - Yan • Unsupervised learning (3 sessions) • K-means clustering • DBSCAN - Cheng • Mean shift • Agglomerative clustering – Kunal • Spectral clustering – Yan • Dimension reduction for data visualization - Yan • Deep learning • Neural network - Yan • Convolutional neural network – Hengyang Lu • Recurrent neural networks – Yan • Hands-on session with deep nets - Yan Slides posted on: http://www.slideshare.net/xuyangela
- 4. More deep learningcoming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase
- 5. Outline • Gradient Descent •Stochastic Gradient Descent (SGD) • Variants of SGD • Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam)
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12. SGD recommendation • Randomlyshuffle training samples • Monitor training and validation error • Experiment learning rates using small sample of training set • Leverage sparsity of training samples • Varying learning rate:
- 13. Variants of SGD •Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam) Ref: https://moodle2.cs.huji.ac.il/nu15/pluginfile.php/316969/mod_resource/conte nt/1/adam_pres.pdf
- 14.
- 15.
- 16.
- 17. The momentum methodby Dr. Geoffrey Hinton https://www.youtube.com/watch?v=LdkkZglLZ0Q&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=27
- 18. SGD with momentum Startwith 0.5
- 19. NAG (Nesterov’s AcceleratedGradient)
- 20. AdaGrad Adaptive learning rate: •weights that receive high gradients will have their effective learning rate reduced • weights that receive small or infrequent updates will have their effective learning rate increased
- 21.
- 22.
- 23.
- 24. Comparisons of DifferentOptimization Methods
- 25. MINIST Comparisons of DifferentOptimization Methods
- 26. CIFAR-10 Comparisons of DifferentOptimization Methods
- 27. Summary of learningmethods for DL https://www.youtube.com/watch?v=defQQqkXEfE&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=29 from:7:33
- 28. Try it out! Fromhands-on session: https://www.dropbox.com/s/92sckhnf1hjgjlo/CNN.zip?dl=0 model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=input_shape)) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) ……. model.add(Flatten()) model.add(Dense(64)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(1)) model.add(Activation('sigmoid')) model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) Optimizer: SGD, RMSprop, Adagrad, Adam…. (https://keras.io/optimizers/)
- 29.
- 30. More deep learningcoming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase • Any proposal?
- 31. Thank you Slides willbe posted at: http://www.slideshare.net/xuyangela Leave a group review please
Editor's Notes
- #8 Perpendicular contour