Optimizers

optimizer ?
2018. 12. 15.

MODUCON 2018

Modulabs Research Scientist

Il Gu Yi

2018.12.15. MODUCON
• Research Scientist

• Ph. D. in Physics

• Research interests

• Generative models (GANs)

• Style transfer

• Reinforcement learning

• Generate and Transfer for Art (GTA Lab)

• github: https://github.com/ilguyi

• e-mail: ilgu.yi@modulabs.co.kr

?
2018. 12. 15.

MODUCON 2018

Modulabs Research Scientist

Il Gu Yi

2018.12.15. MODUCON
Contents
• Introduction to Optimization

• Gradient Descent

• Momentum

• Adaptive Learning Rates

• Adagrad

• RMSprop

• Adam

Optimization &
Gradient Descent

2018.12.15. MODUCON
Optimization
• Optimization problem

• : optimization variables

• : objective function

• : constraint functions
x = (x1, · · · , xn)<latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit><latexit sha1_base64="0C/2KJqMw1f6HmXzEfVk9MaQQaE=">AAACA3icbZDLSgMxFIYzXmu9jbrTTbAIFUqZEUE3QsFNlxXsBdphyGTSNjSTDElGWoYBN76KGxeKuPUl3Pk2pu0stPWHwJf/nENy/iBmVGnH+bZWVtfWNzYLW8Xtnd29ffvgsKVEIjFpYsGE7ARIEUY5aWqqGenEkqAoYKQdjG6n9fYDkYoKfq8nMfEiNOC0TzHSxvLt4zG8geWxn7pZBfZwKLSqQHPl2blvl5yqMxNcBjeHEsjV8O2vXihwEhGuMUNKdV0n1l6KpKaYkazYSxSJER6hAeka5CgiyktnO2TwzDgh7AtpDtdw5v6eSFGk1CQKTGeE9FAt1qbmf7VuovvXXkp5nGjC8fyhfsKgFnAaCAypJFiziQGEJTV/hXiIJMLaxFY0IbiLKy9D66LqGr67LNXqeRwFcAJOQRm44ArUQB00QBNg8AiewSt4s56sF+vd+pi3rlj5zBH4I+vzB64Blkk=</latexit>
f0 : Rn
! R<latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit><latexit sha1_base64="OZomHntDYhKMKYyewg3gP9oXwoE=">AAACFnicbZDLSsNAFIYnXmu9RV26GSyCG0siguKq4KbLKvYCTQyT6aQdOpkJMxOlhDyFG1/FjQtF3Io738ZJ20VtPTDw8f/nMOf8YcKo0o7zYy0tr6yurZc2yptb2zu79t5+S4lUYtLEggnZCZEijHLS1FQz0kkkQXHISDscXhd++4FIRQW/06OE+DHqcxpRjLSRAvs0CjInh1fQi5EehGF2m99nPIeepP2BRlKKxxkrsCtO1RkXXAR3ChUwrUZgf3s9gdOYcI0ZUqrrOon2MyQ1xYzkZS9VJEF4iPqka5CjmCg/G5+Vw2Oj9GAkpHlcw7E6O5GhWKlRHJrOYkM17xXif1431dGln1GepJpwPPkoShnUAhYZwR6VBGs2MoCwpGZXiAdIIqxNkmUTgjt/8iK0zqqu4ZvzSq0+jaMEDsEROAEuuAA1UAcN0AQYPIEX8AberWfr1fqwPietS9Z05gD8KevrF5TUn6s=</latexit>
Slide credit: Boyd & Vandenberghe, https://web.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf, p. 2
minimize f0(x)
subject to fi(x)  0, i = {1, · · · , m}
hj(x) = 0, j = {1, · · · , l}<latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit><latexit sha1_base64="KC9yMOES57E4+eIe5xM1pVtoKo4=">AAACjnicbVFda9swFJW9ry7dh7c97uWysNJBKXYZ2+gILewljx0sbSEKRpavE6WS7ElyaWb8c/qH9rZ/M8UxY0t3QXA451zp3qOsksK6OP4VhPfuP3j4aOfxYPfJ02fPoxcvz21ZG44TXsrSXGbMohQaJ044iZeVQaYyiRfZ1Ze1fnGNxopSf3OrCmeKzbUoBGfOU2l0SzOcC90wY9iqbYxsB9ThjbNFo4QWSvzAFvagSJu43b95R+kf2dbZErkDV/YGsTYAlfgd4gOgIGAEtEk85Hnp7AEo2vr+PVikzbLzjjbG5bZRdkaKOu/nSqNhfBh3BXdB0oMh6essjX7SvOS1Qu24ZNZOk7hyM3+bE1yi37G2WDF+xeY49VAzhXbWdHG28NYzORSl8Uc76Ni/OxqmrF2pzDsVcwu7ra3J/2nT2hWfZo3QVe1Q881DRS19hLD+G8iF8YnKlQeMG+FnBb5ghnHnf3DgQ0i2V74Lzo8OE4+/vh+ejvs4dshr8obsk4R8JKdkTM7IhPBgN0iC4+BzGIUfwlF4srGGQd/zivxT4fg3OLPCKw==</latexit>
fi, hj : Rn
! R<latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit><latexit sha1_base64="ANhN+8q3qLNgLotTf4cOApMtiwo=">AAACHXicbZDLSsNAFIYnXmu9RV26GSyCCymJFBRXBTddVrEXaGOYTCfN2MkkzEyUEvIibnwVNy4UceFGfBsnbRa19cDAx/+fw5zzezGjUlnWj7G0vLK6tl7aKG9ube/smnv7bRklApMWjlgkuh6ShFFOWooqRrqxICj0GOl4o6vc7zwQIWnEb9U4Jk6Ihpz6FCOlJdes+W5Ks1MYuOl9Bi9hP0Qq8Lz0JrtLeQb7gg4DhYSIHmcs16xYVWtScBHsAiqgqKZrfvUHEU5CwhVmSMqebcXKSZFQFDOSlfuJJDHCIzQkPY0chUQ66eS6DB5rZQD9SOjHFZyosxMpCqUch57uzDeU814u/uf1EuVfOCnlcaIIx9OP/IRBFcE8KjiggmDFxhoQFlTvCnGABMJKB1rWIdjzJy9C+6xqa76uVeqNIo4SOARH4ATY4BzUQQM0QQtg8ARewBt4N56NV+PD+Jy2LhnFzAH4U8b3LyrFop8=</latexit>
Figure credit: http://neuralnetworksanddeeplearning.com/chap2.html

2018.12.15. MODUCON
Examples
• Portfolio optimization
• variables: amounts invested in different assets

• constraints: budget, max./min. investment per asset, minimum return

• objective: overall risk or return variance

• Device sizing in electronic circuits
• variables: device widths and lengths

• constraints: manufacturing limits, timing requirements, maximum area

• objective: power consumption

• Data fitting
• variables: model parameters

• constraints: prior information, parameter limits

• objective: measure of misfit or prediction error
Slide credit: Boyd & Vandenberghe, https://web.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf, p. 3

2018.12.15. MODUCON
Why do We Care?
• Optimization is at the heart of many (most practical?) machine learning
algorithms

• Linear regression:

• Classiﬁcation (logistic regression or SVM):
minimize
w
||Xw y||2
<latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit>
minimize
w
nX
i=1
log(1 + exp( yix>
i w))
<latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit>
or ||w||2
+ C
nX
i=1
⇠i s.t. ⇠i  1 yix>
i w, ⇠i  0
<latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit>
Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 5

2018.12.15. MODUCON
We still Care…
• Maximum likelihood estimation:

• Collaborative ﬁltering:

• k-means:

• And more (graphical models, feature selection, active learning, control)
Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 6
maximize
✓
nX
i=1
log p✓(xi)
<latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit>
minimize
w
X
i j
log 1 + exp(w>
xi w>
xj)
<latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit>
minimize
µ1,··· ,µk
J(µ) =
kX
j=1
X
i2Cj
||xi µj||2
<latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit>

2018.12.15. MODUCON
Optimization Algorithm
• Optimization problem

• Just iterate (until convergence)
Figure credit: http://neuralnetworksanddeeplearning.com/chap2.html
minimize
x
f0(x)
<latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit><latexit sha1_base64="SUKSt8CO6vpen+Yzt4cG9g8bKc8=">AAACE3icbZA9SwNBEIb3/IzxK2ppsxiEaCF3ImgZsEkZwUQhF8LeZk4Xd/eO3TlJPO4/2PhXbCwUsbWx89+4iSk0OrDw8L4zzM4bpVJY9P1Pb2Z2bn5hsbRUXl5ZXVuvbGy2bZIZDi2eyMRcRsyCFBpaKFDCZWqAqUjCRXRzOvIvbsFYkehzHKbQVexKi1hwhk7qVfbDTPedD5gPijxEGKCNcyW0UOIOioKGNO7lflEb7PUqVf/AHxf9C8EEqmRSzV7lI+wnPFOgkUtmbSfwU+zmzKDgEopymFlIGb9hV9BxqJkC283HNxV01yl9GifGPY10rP6cyJmydqgi16kYXttpbyT+53UyjE+6udBphqD596I4kxQTOgqI9oUBjnLogHEj3F8pv2aGcXQxlV0IwfTJf6F9eBA4Pjuq1huTOEpkm+yQGgnIMamTBmmSFuHknjySZ/LiPXhP3qv39t06401mtsiv8t6/AIxgnyk=</latexit>
xt+1 = xt ⌘t
@f0(xt)
@xt<latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit><latexit sha1_base64="TwFLJIBhxE8d1aAfdQ3ecuCuni4=">AAACMHicbZDLSgMxFIYzXmu9VV26CRahIpYZEXQjFFzYZQV7gU4ZMmmmDc1cSM6IZZhHcuOj6EZBEbc+hZlpQW09EPj4/3OSnN+NBFdgmq/GwuLS8spqYa24vrG5tV3a2W2pMJaUNWkoQtlxiWKCB6wJHATrRJIR3xWs7Y6uMr99x6TiYXAL44j1fDIIuMcpAS05pet7J4FjK8WXOKMUn2CbAcnR9iShiR0RCZwI7DmJmVbyrqP0R86F1CmVzaqZF54HawplNK2GU3qy+yGNfRYAFUSprmVG0EuyS6lgadGOFYsIHZEB62oMiM9UL8kXTvGhVvrYC6U+AeBc/T2REF+pse/qTp/AUM16mfif143Bu+glPIhiYAGdPOTFAkOIs/Rwn0tGQYw1ECq5/iumQ6JjAp1xUYdgza48D63TqqX55qxcq0/jKKB9dIAqyELnqIbqqIGaiKIH9Ize0LvxaLwYH8bnpHXBmM7soT9lfH0D/MuqQg==</latexit>

2018.12.15. MODUCONFigure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 111

2018.12.15. MODUCON
initial value
t = 0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
Gradient Descent

2018.12.15. MODUCON
gradient
Gradient Descent

2018.12.15. MODUCON
update
✓0<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
Gradient Descent

2018.12.15. MODUCON
update
Gradient Descent

2018.12.15. MODUCON
gradient
Gradient Descent

2018.12.15. MODUCON
update
Gradient Descent

2018.12.15. MODUCON
gradient
Gradient Descent

2018.12.15. MODUCON
update
Gradient Descent

2018.12.15. MODUCON
gradient = 0
Gradient Descentt = 3<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

2018.12.15. MODUCON
gradient = 0
No more update
t = 3, 4, · · ·<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
wt+1 = wt ⌘
@L(wt)
@wt<latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit><latexit sha1_base64="0Nq+MOAB0QsY2AM8V/2/XqJNUBo=">AAACWXicbVFNSwMxEM2uX3X9qvboJVgURZRdEfQiFLz04KGCbYVuKbNpVoPZD5JZpSz7Jz0I4l/xYLYtYlsHAo/33kwmL0EqhUbX/bTspeWV1bXKurOxubW9U93d6+gkU4y3WSIT9RiA5lLEvI0CJX9MFYcokLwbvNyWeveVKy2S+AFHKe9H8BSLUDBAQw2qqR8BPgdh/lYMcjz1Cnp0Q2e4gp5RnyNQP1TAcj8FhQLkxMRA5nfF8WzDSTHn+lUKxxlU6+65Oy66CLwpqJNptQbVd3+YsCziMTIJWvc8N8V+Xs5nkheOn2meAnuBJ94zMIaI634+Tqagh4YZ0jBR5sRIx+zfjhwirUdRYJzlonpeK8n/tF6G4XU/F3GaIY/Z5KIwkxQTWsZMh0JxhnJkADAlzK6UPYMJEM1nlCF4809eBJ2Lc8/g+8t6ozmNo0L2yQE5Jh65Ig3SJC3SJox8kG9rxVq1vmzLrtjOxGpb054amSm79gNNR7Xy</latexit>
Gradient Descent

2018.12.15. MODUCON
Computing the Gradient
• Use backpropagation to compute gradients efficiently

• Need a differentiable function

• Can’t use functions like argmax or hard binary

• Unless using a different way to compute gradients
Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19

2018.12.15. MODUCON
How to Pick the Learning Rate?
• Too big = diverge, too small = slow convergence

• No “one learning rate to rule them all”

• Start from a high value and keep cutting by half if model diverges

• Learning rate schedule: decay learning rate over time
Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19

2018.12.15. MODUCON
Too Small Learning Rate
Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112

2018.12.15. MODUCON
Too Large Learning Rate
Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112

2018.12.15. MODUCON
Learning Rate
• Which is better?

• Is it better to keep learning rate?

• Decay learning rate
appropriately
Figure credit: cs231n spring 2018 slide: Lecture 6. p. 84

2018.12.15. MODUCON
Stochastic Gradient Descent

• Cross entropy error, CEE
w0
k := wk ⌘
@L
@wk
• Loss
(mini-batch)
Loss

• Mini-batch size:

•
L
1
N i
yi log ˆyi
L
1
m i
yi log ˆyi
m<latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit>

2018.12.15. MODUCON
Gradient Descent Pitfalls I

2018.12.15. MODUCON
The Momentum Method
• Introduce velocity variable:

• It is the direction and speed at which parameters move through parameter
space

• Momentum is mass times velocity term in physics

• The momentum algorithm assumes unit mass

• A hyperparameter determines exponential decay
v<latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit>
2 [0, 1)<latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit>

2018.12.15. MODUCON
The Momentum Method
wt+1 = wt ⌘
@L(wt)
SGD SGD + Momentum
vt+1 = vt ⌘
@L(wt)
@wt
wt+1 = wt + vt+1<latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit><latexit sha1_base64="U3U4Wtcm56nrU4SV6GNYM7/CY+0=">AAAClnicbVFdSxtBFJ3d2hrTD6O+CL4MDS0pobIrheqDIoqYBx9SaD4gG8LdyWwcnNldZu6mhGV/kn/GN/+Ns0kUE3th4HDOudwz94apFAY979Fx3228/7BZ2ap+/PT5y3ZtZ7drkkwz3mGJTHQ/BMOliHkHBUreTzUHFUreC+8uS7035dqIJP6Ls5QPFUxiEQkGaKlR7T5QgLdhlE+LUY5Nv6DfT2kwAaWArkgF/UkDjpaNNLA8SEGjALkwMZD5TdF4bvg3b/hRrLlelCIIqivM89wVF23S9XCjWt079OZF3wJ/CepkWe1R7SEYJyxTPEYmwZiB76U4zMtUTPKiGmSGp8DuYMIHFsaguBnm87UW9JtlxjRKtH0x0jn7uiMHZcxMhdZZ5jTrWkn+TxtkGB0PcxGnGfKYLQZFmaSY0PJGdCw0ZyhnFgDTwmal7Bbs2tFesmqX4K9/+S3oHh36Fv/5VT9vLddRIQfkK2kQn/wm56RF2qRDmLPnnDgXzqW77565V+71wuo6y549slJu+wmjPMxf</latexit>
SGD + Momentum another form
vt+1 = vt +
@L(wt)
@wt
wt+1 = wt ⌘vt+1<latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit><latexit sha1_base64="oHZOwYQyz2UgHSDkKdJ/c19tcsI=">AAACmnicbVFNa9tAEF2paZs6/VCbQw7JYalpcTENUikkF4NJoE1IDynYScBrzGi9cpbsSmJ3lGCEflT+Sm/9N13ZTojtDiw83nvDvJ2JcyUthuFfz3+28fzFy81Xja3Xb96+C95/uLBZYbjo80xl5ioGK5RMRR8lKnGVGwE6VuIyvjmu9ctbYazM0h5OczHUMEllIjmgo0bBPdOA13FS3lajEttRRT93KJuA1kCXpIq2KUsM8JLlYFCCmuscVPmraj1472beL9WK61GpGGssMQ8jl1z0K2UCVxI4p2tujIJmuB/Oiq6DaAGaZFHno+APG2e80CJFrsDaQRTmOCzreFyJqsEKK3LgNzARAwdT0MIOy9lqK/rJMWOaZMa9FOmMfdpRgrZ2qmPnrLPaVa0m/6cNCkwOh6VM8wJFyueDkkJRzGh9JzqWRnBUUweAG+myUn4Nbv/orlkvIVr98jq4+LYfOfz7e7N7sljHJtklH0mLROSAdMkJOSd9wr0dr+P98H76e/6Rf+qfza2+t+jZJkvl9/4BkujNUw==</latexit>

2018.12.15. MODUCON
Pseudo Codes
wt+1 = wt ⌘
@L(wt)
SGD SGD + Momentum
vt+1 = vt ⌘
@L(wt)
@wt
for step in range(max_steps):
grads = gradients(vars)
vars = vars - lr * grads
velocity = 0.0
for step in range(max_steps):
grads = gradients(vars)
velocity = momentum * velocity -
lr * grads
vars = vars + velocity

2018.12.15. MODUCON
Gradient Descent

2018.12.15. MODUCON
gradient
Gradient Descent

2018.12.15. MODUCON
update
Gradient Descent

2018.12.15. MODUCON
update

2018.12.15. MODUCON
gradient

2018.12.15. MODUCON
update

2018.12.15. MODUCON
gradient = 0
when SGD

2018.12.15. MODUCON
gradient
velocity
GD with Momentumt = 1<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

2018.12.15. MODUCON
gradient
velocity
update = velocity + gradient

2018.12.15. MODUCON
gradient
velocity
vt+1 = vt ⌘
@L(wt)
@wt

2018.12.15. MODUCON
gradient
velocity
Compare with GDt = 2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
vt+1 = vt ⌘
@L(wt)
@wt
✓GD
2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

2018.12.15. MODUCONFigure credit: cs231n spring 2018 slide: Lecture 7. p. 31

2018.12.15. MODUCON
Gradient Descent Pitfalls II
Figure credit: cs231n spring 2018 slide: Lecture 7. p. 22
Figure credit: https://www.willamette.edu/~gorr/classes/cs449/momrate.html
SGD without momentum
SGD with momentum

2018.12.15. MODUCON
Path of Gradient Descent
Figure credit: https://github.com/ilguyi/optimizers.numpy

2018.12.15. MODUCON
Path of GD with Momentum
Figure credit: https://github.com/ilguyi/optimizers.numpy

Algorithms of Adaptive Learning
Rates

2018.12.15. MODUCON
Learning Rate is Crucial
• Learning rate: most difficult hyperparameters to set

• It significantly affects model performance

• Loss function is highly sensitive to some directions in parameter space and
insensitive to others

• Momentum helps but introduces another hyperparameters

• If direction of sensitivity is axis aligned, separate learning rate for each
parameter and adjust them throughput learning

2018.12.15. MODUCON
Recent Algorithms
• Adagrad

• RMSprop

• Adam

• AdaMax

• NAdam

2018.12.15. MODUCON
Adagrad
• J. Duchi, et. al., Adaptive subgradient methods for online learning and
stochastic optimization (http://jmlr.org/papers/v12/duchi11a.html)

• It adapts the learning rate to the parameters, performing smaller updates (i.e.
low learning rates) for parameters associated with frequently occurring
features, and larger updates (i.e. high learning rates) for parameters
associated with infrequent features

• Previously, we performed an update for all parameters at once as every
parameter used the same learning rate

• As Adagrad uses a diﬀerent learning rate for every parameter at every time
step
w<latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit>
wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit>
⌘<latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit>
wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit>
t<latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit>

2018.12.15. MODUCON
Adagrad
✓0
1 = ✓1 ⌘1
@L
@✓1
✓0
2 = ✓2 ⌘2
@L
@✓2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
Gradient Descent
⌘1 = ⌘2<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
where,

Optimizers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizers

Similar to Optimizers (8)

Recently uploaded

Recently uploaded (20)

Optimizers