Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Optimizers Slide 1 Optimizers Slide 2 Optimizers Slide 3 Optimizers Slide 4 Optimizers Slide 5 Optimizers Slide 6 Optimizers Slide 7 Optimizers Slide 8 Optimizers Slide 9 Optimizers Slide 10 Optimizers Slide 11 Optimizers Slide 12 Optimizers Slide 13 Optimizers Slide 14 Optimizers Slide 15 Optimizers Slide 16 Optimizers Slide 17 Optimizers Slide 18 Optimizers Slide 19 Optimizers Slide 20 Optimizers Slide 21 Optimizers Slide 22 Optimizers Slide 23 Optimizers Slide 24 Optimizers Slide 25 Optimizers Slide 26 Optimizers Slide 27 Optimizers Slide 28 Optimizers Slide 29 Optimizers Slide 30 Optimizers Slide 31 Optimizers Slide 32 Optimizers Slide 33 Optimizers Slide 34 Optimizers Slide 35 Optimizers Slide 36 Optimizers Slide 37 Optimizers Slide 38 Optimizers Slide 39 Optimizers Slide 40 Optimizers Slide 41 Optimizers Slide 42 Optimizers Slide 43 Optimizers Slide 44 Optimizers Slide 45 Optimizers Slide 46 Optimizers Slide 47 Optimizers Slide 48 Optimizers Slide 49 Optimizers Slide 50 Optimizers Slide 51 Optimizers Slide 52 Optimizers Slide 53 Optimizers Slide 54 Optimizers Slide 55 Optimizers Slide 56 Optimizers Slide 57 Optimizers Slide 58 Optimizers Slide 59 Optimizers Slide 60 Optimizers Slide 61 Optimizers Slide 62 Optimizers Slide 63 Optimizers Slide 64 Optimizers Slide 65 Optimizers Slide 66 Optimizers Slide 67 Optimizers Slide 68 Optimizers Slide 69 Optimizers Slide 70 Optimizers Slide 71 Optimizers Slide 72 Optimizers Slide 73 Optimizers Slide 74 Optimizers Slide 75 Optimizers Slide 76 Optimizers Slide 77 Optimizers Slide 78 Optimizers Slide 79
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Optimizers

Download to read offline

I have implemented various optimizers (gradient descent, momentum, adam, etc.) based on gradient descent using only numpy not deep learning framework like TensorFlow.

Related Books

Free with a 30 day trial from Scribd

See all

Optimizers

  1. 1. optimizer ? 2018. 12. 15. MODUCON 2018 Modulabs Research Scientist Il Gu Yi
  2. 2. 2018.12.15. MODUCON • Research Scientist • Ph. D. in Physics • Research interests • Generative models (GANs) • Style transfer • Reinforcement learning • Generate and Transfer for Art (GTA Lab) • github: https://github.com/ilguyi • e-mail: ilgu.yi@modulabs.co.kr
  3. 3. ? 2018. 12. 15. MODUCON 2018 Modulabs Research Scientist Il Gu Yi
  4. 4. 2018.12.15. MODUCON
  5. 5. 2018.12.15. MODUCON Contents • Introduction to Optimization • Gradient Descent • Momentum • Adaptive Learning Rates • Adagrad • RMSprop • Adam
  6. 6. Optimization & Gradient Descent
  7. 7. 2018.12.15. MODUCON Optimization • Optimization problem • : optimization variables • : objective function • : constraint functions x = (x1, · · · , xn)<latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit> f0 : Rn ! R<latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit> Slide credit: Boyd & Vandenberghe, https://web.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf, p. 2 minimize f0(x) subject to fi(x)  0, i = {1, · · · , m} hj(x) = 0, j = {1, · · · , l}<latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit> fi, hj : Rn ! R<latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit> Figure credit: http://neuralnetworksanddeeplearning.com/chap2.html
  8. 8. 2018.12.15. MODUCON Examples • Portfolio optimization • variables: amounts invested in different assets • constraints: budget, max./min. investment per asset, minimum return • objective: overall risk or return variance • Device sizing in electronic circuits • variables: device widths and lengths • constraints: manufacturing limits, timing requirements, maximum area • objective: power consumption • Data fitting • variables: model parameters • constraints: prior information, parameter limits • objective: measure of misfit or prediction error Slide credit: Boyd & Vandenberghe, https://web.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf, p. 3
  9. 9. 2018.12.15. MODUCON Why do We Care? • Optimization is at the heart of many (most practical?) machine learning algorithms • Linear regression: • Classification (logistic regression or SVM): minimize w ||Xw y||2 <latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit> minimize w nX i=1 log(1 + exp( yix> i w)) <latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit> or ||w||2 + C nX i=1 ⇠i s.t. ⇠i  1 yix> i w, ⇠i  0 <latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit> Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 5
  10. 10. 2018.12.15. MODUCON We still Care… • Maximum likelihood estimation: • Collaborative filtering: • k-means: • And more (graphical models, feature selection, active learning, control) Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 6 maximize ✓ nX i=1 log p✓(xi) <latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit> minimize w X i j log 1 + exp(w> xi w> xj) <latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit> minimize µ1,··· ,µk J(µ) = kX j=1 X i2Cj ||xi µj||2 <latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit>
  11. 11. 2018.12.15. MODUCON Optimization Algorithm • Optimization problem • Gradient Descent • Just iterate (until convergence) Figure credit: http://neuralnetworksanddeeplearning.com/chap2.html minimize x f0(x) <latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit> xt+1 = xt ⌘t @f0(xt) @xt<latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit>
  12. 12. 2018.12.15. MODUCONFigure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 111
  13. 13. 2018.12.15. MODUCON initial value t = 0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  14. 14. 2018.12.15. MODUCON gradient t = 0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  15. 15. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  16. 16. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  17. 17. 2018.12.15. MODUCON gradient ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  18. 18. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  19. 19. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  20. 20. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient t = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  21. 21. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  22. 22. 2018.12.15. MODUCON update ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  23. 23. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient = 0 Gradient Descentt = 3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  24. 24. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient = 0 No more update t = 3, 4, · · ·<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> wt+1 = wt ⌘ @L(wt) @wt<latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit> Gradient Descent
  25. 25. 2018.12.15. MODUCON Computing the Gradient • Use backpropagation to compute gradients efficiently • Need a differentiable function • Can’t use functions like argmax or hard binary • Unless using a different way to compute gradients Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19
  26. 26. 2018.12.15. MODUCON How to Pick the Learning Rate? • Too big = diverge, too small = slow convergence • No “one learning rate to rule them all” • Start from a high value and keep cutting by half if model diverges • Learning rate schedule: decay learning rate over time Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19
  27. 27. 2018.12.15. MODUCON Too Small Learning Rate Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112
  28. 28. 2018.12.15. MODUCON Too Large Learning Rate Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112
  29. 29. 2018.12.15. MODUCON Learning Rate • Which is better? • Is it better to keep learning rate? • Decay learning rate appropriately Figure credit: cs231n spring 2018 slide: Lecture 6. p. 84
  30. 30. 2018.12.15. MODUCON Stochastic Gradient Descent • Gradient Descent • Cross entropy error, CEE w0 k := wk ⌘ @L @wk • Loss (mini-batch) Loss • Mini-batch size: • L 1 N i yi log ˆyi L 1 m i yi log ˆyi m<latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit>
  31. 31. 2018.12.15. MODUCON Gradient Descent Pitfalls I
  32. 32. 2018.12.15. MODUCON The Momentum Method • Introduce velocity variable: • It is the direction and speed at which parameters move through parameter space • Momentum is mass times velocity term in physics • The momentum algorithm assumes unit mass • A hyperparameter determines exponential decay v<latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit> 2 [0, 1)<latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit>
  33. 33. 2018.12.15. MODUCON The Momentum Method wt+1 = wt ⌘ @L(wt) @wt<latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit> SGD SGD + Momentum vt+1 = vt ⌘ @L(wt) @wt wt+1 = wt + vt+1<latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit> SGD + Momentum another form vt+1 = vt + @L(wt) @wt wt+1 = wt ⌘vt+1<latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit>
  34. 34. 2018.12.15. MODUCON Pseudo Codes wt+1 = wt ⌘ @L(wt) @wt<latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit> SGD SGD + Momentum vt+1 = vt ⌘ @L(wt) @wt wt+1 = wt + vt+1<latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit> for step in range(max_steps): grads = gradients(vars) vars = vars - lr * grads velocity = 0.0 for step in range(max_steps): grads = gradients(vars) velocity = momentum * velocity - lr * grads vars = vars + velocity
  35. 35. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient = 0 No more update t = 3, 4, · · ·<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> wt+1 = wt ⌘ @L(wt) @wt<latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit> Gradient Descent
  36. 36. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> t = 0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  37. 37. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient t = 0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  38. 38. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> update t = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent
  39. 39. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> update ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descentt = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  40. 40. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient Gradient Descentt = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  41. 41. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> update Gradient Descentt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  42. 42. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> update ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descentt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  43. 43. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient = 0 when SGD Gradient Descentt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  44. 44. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient velocity GD with Momentumt = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  45. 45. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient velocity update = velocity + gradient GD with Momentumt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  46. 46. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient velocity update = velocity + gradient GD with Momentumt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> vt+1 = vt ⌘ @L(wt) @wt wt+1 = wt + vt+1<latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit>
  47. 47. 2018.12.15. MODUCON ✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> ✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> gradient velocity update = velocity + gradient Compare with GDt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> vt+1 = vt ⌘ @L(wt) @wt wt+1 = wt + vt+1<latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit> ✓GD 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
  48. 48. 2018.12.15. MODUCONFigure credit: cs231n spring 2018 slide: Lecture 7. p. 31
  49. 49. 2018.12.15. MODUCON Gradient Descent Pitfalls II Figure credit: cs231n spring 2018 slide: Lecture 7. p. 22 Figure credit: https://www.willamette.edu/~gorr/classes/cs449/momrate.html SGD without momentum SGD with momentum
  50. 50. 2018.12.15. MODUCON Path of Gradient Descent Figure credit: https://github.com/ilguyi/optimizers.numpy
  51. 51. 2018.12.15. MODUCON Path of GD with Momentum Figure credit: https://github.com/ilguyi/optimizers.numpy
  52. 52. Algorithms of Adaptive Learning Rates
  53. 53. 2018.12.15. MODUCON Learning Rate is Crucial • Learning rate: most difficult hyperparameters to set • It significantly affects model performance • Loss function is highly sensitive to some directions in parameter space and insensitive to others • Momentum helps but introduces another hyperparameters • If direction of sensitivity is axis aligned, separate learning rate for each parameter and adjust them throughput learning
  54. 54. 2018.12.15. MODUCON Recent Algorithms • Adagrad • RMSprop • Adam • AdaMax • NAdam
  55. 55. 2018.12.15. MODUCON Adagrad • J. Duchi, et. al., Adaptive subgradient methods for online learning and stochastic optimization (http://jmlr.org/papers/v12/duchi11a.html) • It adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features • Previously, we performed an update for all parameters at once as every parameter used the same learning rate • As Adagrad uses a different learning rate for every parameter at every time step w<latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit> wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit> ⌘<latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit> wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit> t<latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit>
  56. 56. 2018.12.15. MODUCON Adagrad ✓0 1 = ✓1 ⌘1 @L @✓1 ✓0 2 = ✓2 ⌘2 @L @✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> Gradient Descent ⌘1 = ⌘2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit> where,
  • WooSeongJo1

    Dec. 18, 2018

I have implemented various optimizers (gradient descent, momentum, adam, etc.) based on gradient descent using only numpy not deep learning framework like TensorFlow.

Views

Total views

348

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

12

Shares

0

Comments

0

Likes

1

×