Download as PDF, PPTX
Uploaded byIlya Kuzovkin
Paper overview: "Deep Residual Learning for Image Recognition"
The document discusses deep residual learning for image recognition, detailing the evolution of neural network architectures and their performance on various benchmarks such as ILSVRC and MS COCO. It emphasizes the challenges of vanishing gradients and degradation problems with increased depth, proposing identity connections to mitigate these issues. The findings indicate that ResNet architectures, particularly deeper variants, can achieve lower error rates and improved training accuracy by addressing the degradation problem effectively.
More Related Content
![[DSC 2016] 系列活動：李宏毅 / 一天搞懂深度學習](https://cdn.slidesharecdn.com/ss_thumbnails/1-160521014039-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Learning What and Where to Draw (NIPS’16)](https://cdn.slidesharecdn.com/ss_thumbnails/20170120misono-170215035536-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Dl輪読会]bridging the gaps between residual learning, recurrent neural networks...](https://cdn.slidesharecdn.com/ss_thumbnails/dlbridgingthegapsbetweenresiduallearningrecurrentneuralnetworksandvisualcortex-160809011746-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Image-to-Image Translation with Conditional Adversarial Networks](https://cdn.slidesharecdn.com/ss_thumbnails/readingpaper20160113-170222033302-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Dl輪読会]bayesian dark knowledge](https://cdn.slidesharecdn.com/ss_thumbnails/dlbayesiandarkknowledge-160819032303-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generat...](https://cdn.slidesharecdn.com/ss_thumbnails/dlhacks170303-170308031242-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Regularization with stochastic transformations and perturbations for d...](https://cdn.slidesharecdn.com/ss_thumbnails/regularizationwithstochastictransformationsandperturbationsfordeepsemi-supervisedlearning-170215034444-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Paper overview: "Deep Residual Learning for Image Recognition"
Paper overview: "Deep Residual Learning for Image Recognition"
- 1. Article overview by IlyaKuzovkin K. He, X. Zhang, S. Ren and J. Sun Microsoft Research Computational Neuroscience Seminar University of Tartu 2016 Deep Residual Learning for Image Recognition ILSVRC 2015 MS COCO 2015 WINNER
- 2.
- 3.
- 4.
- 5. 2012 8 layers 15.31% error 2013 9layers, 2x params 11.74% error
- 6. 2012 8 layers 15.31% error 20132014 9 layers, 2x params 11.74% error 19 layers 7.41% error
- 7. 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error ?
- 8. 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error Is learning better networks as easy as stacking more layers? ?
- 9. ? 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error Is learning better networks as easy as stacking more layers? Vanishing / exploding gradients
- 10. ? 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error Is learning better networks as easy as stacking more layers? Vanishing / exploding gradients Normalized initialization & intermediate normalization
- 11. ? 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error Is learning better networks as easy as stacking more layers? Vanishing / exploding gradients Normalized initialization & intermediate normalization Degradation problem
- 12. Degradation problem “with thenetwork depth increasing, accuracy gets saturated”
- 13. Not caused byoverfitting: Degradation problem “with the network depth increasing, accuracy gets saturated”
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Conv Conv Conv Conv Trained Accuracy X% Conv Conv Conv Conv Identity Identity Identity Identity Same performance Conv Conv Conv Conv Conv Conv Conv Conv Trained Worse! Tested Tested Tested “Our currentsolvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time)”
- 20. Conv Conv Conv Conv Trained Accuracy X% Conv Conv Conv Conv Identity Identity Identity Identity Same performance Conv Conv Conv Conv Conv Conv Conv Conv Trained Worse! Tested Tested Tested “Our currentsolvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time)” “Solvers might have difficulties in approximating identity mappings by multiple nonlinear layers”
- 21. Conv Conv Conv Conv Trained Accuracy X% Conv Conv Conv Conv Identity Identity Identity Identity Same performance Conv Conv Conv Conv Conv Conv Conv Conv Trained Worse! Tested Tested Tested “Our currentsolvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time)” “Solvers might have difficulties in approximating identity mappings by multiple nonlinear layers” Add explicit identity connections and “solvers may simply drive the weights of the multiple nonlinear layers toward zero”
- 22. Add explicit identityconnections and “solvers may simply drive the weights of the multiple nonlinear layers toward zero” is the true function we want to learn
- 23. Add explicit identityconnections and “solvers may simply drive the weights of the multiple nonlinear layers toward zero” is the true function we want to learn Let’s pretend we want to learn instead.
- 24. Add explicit identityconnections and “solvers may simply drive the weights of the multiple nonlinear layers toward zero” is the true function we want to learn Let’s pretend we want to learn instead. The original function is then
- 26. Network can decidehow deep it needs to be…
- 27. Network can decidehow deep it needs to be… “The identity connections introduce neither extra parameter nor computation complexity”
- 28. 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error ?
- 29. 2012 8 layers 15.31% error 20132014 2015 9 layers, 2x params 11.74% error 19 layers 7.41% error 152 layers 3.57% error
- 30.
- 31. • Lots ofconvolutional 3x3 layers • VGG complexity is 19.6 billion FLOPs 34-layer-ResNet is 3.6 bln. FLOPs
- 32. • Lots ofconvolutional 3x3 layers • VGG complexity is 19.6 billion FLOPs 34-layer-ResNet is 3.6 bln. FLOPs ! • Batch normalization • SGD with batch size 256 • (up to) 600,000 iterations • LR 0.1 (divided by 10 when error plateaus) • Momentum 0.9 • No dropout • Weight decay 0.0001
- 33. • Lots ofconvolutional 3x3 layers • VGG complexity is 19.6 billion FLOPs 34-layer-ResNet is 3.6 bln. FLOPs ! • Batch normalization • SGD with batch size 256 • (up to) 600,000 iterations • LR 0.1 (divided by 10 when error plateaus) • Momentum 0.9 • No dropout • Weight decay 0.0001 ! • 1.28 million training images • 50,000 validation • 100,000 test
- 34. • 34-layer ResNethas lower training error. This indicates that the degradation problem is well addressed and we manage to obtain accuracy gains from increased depth.
- 35. • 34-layer ResNethas lower training error. This indicates that the degradation problem is well addressed and we manage to obtain accuracy gains from increased depth. ! • 34-layer-ResNet reduces the top-1 error by 3.5%
- 36. • 34-layer ResNethas lower training error. This indicates that the degradation problem is well addressed and we manage to obtain accuracy gains from increased depth. ! • 34-layer-ResNet reduces the top-1 error by 3.5% ! • 18-layer ResNet converges faster and thus ResNet eases the optimization by providing faster convergence at the early stage.
- 37.
- 38. Due to timecomplexity the usual building block is replaced by Bottleneck Block 50 / 101 / 152 - layer ResNets are build from those blocks
- 41.
- 44. ImageNet Classification 20151st 3.57% error ImageNet Object Detection 2015 1st 194 / 200 categories ImageNet Object Localization 2015 1st 9.02% error COCO Detection 2015 1st 37.3% COCO Segmentation 2015 1st 28.2% http://research.microsoft.com/en-us/um/people/kahe/ilsvrc15/ilsvrc2015_deep_residual_learning_kaiminghe.pdf