Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

D1L5 Visualization (D1L2 Insight@DCU Machine Learning Workshop 2017)

263 views

Published on

https://github.com/telecombcn-dl/dlmm-2017-dcu

Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

D1L5 Visualization (D1L2 Insight@DCU Machine Learning Workshop 2017)

  1. 1. Amaia Salvador amaia.salvador@upc.edu PhD Candidate Universitat Politècnica de Catalunya Visualization Day 1 Lecture 5
  2. 2. 2
  3. 3. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 3
  4. 4. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 4
  5. 5. Visualize Learned Weights AlexNet conv1 Filters are only “interpretable” on the first layer 5
  6. 6. Visualize Learned Weights layer 2 weights layer 3 weights Source: ConvnetJS 6
  7. 7. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 7
  8. 8. Activations from data 8 conv1 conv5
  9. 9. Receptive Field Girshick et al. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR 2014 9 “people” “text” Visualize the receptive field of a neuron on those images that activate it the most
  10. 10. Reminder: Receptive Field 10 Receptive field: Part of the input that is visible to a neuron. It increases as we stack more convolutional layers (i.e. neurons in deeper layers have larger receptive fields).
  11. 11. Occlusion experiments 1. Iteratively forward the same image through the network, occluding a different region at a time. 2. Keep track of the probability of the correct class w.r.t. the position of the occluder Zeiler and Fergus. Visualizing and Understanding Convolutional Networks. ECCV 2015 11
  12. 12. Occlusion experiments 12 (any arbitrary layer) Zhou et al. Object detectors emerge in deep scene CNNs. ICLR 2015
  13. 13. Class Activation Maps 13 Class Activation Maps (CAMs): Replace FC layer after last conv with Global Average Pooling (GAP). Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. "Learning deep features for discriminative localization." CVPR 2016 densely connected weights
  14. 14. Class Activation Maps 14 Class Activation Maps (CAMs): The classifier weights define the contribution of each channel in the previous layer Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. "Learning deep features for discriminative localization." CVPR 2016 contribution of the blue channel in the conv layer to the prediction of “australian terrier”
  15. 15. Class Activation Maps 15 Class Activation Maps (CAMs): Unsupervised object localization Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. "Learning deep features for discriminative localization." CVPR 2016
  16. 16. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 16
  17. 17. t-SNE Embed high dimensional data points (i.e. feature codes) so that pairwise distances are preserved in local neighborhoods. Maaten & Hinton. Visualizing High-Dimensional Data using t-SNE. Journal of Machine Learning Research (2008). 17
  18. 18. t-SNE Can be used with features from layer before classification 18
  19. 19. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 19
  20. 20. Gradient-based approach Compute the gradient of any neuron w.r.t. the image 1. Forward image up to the desired layer (e.g. conv5) 2. Set all gradients to 0 3. Set gradient for the neuron we are interested in to 1 4. Backpropagate to get reconstructed image (gradient on the image) Visualize the part of an image that mostly activates a neuron 20
  21. 21. Gradient-based approach 1. Forward image up to the desired layer (e.g. conv5) 2. Set all gradients to 0 3. Set gradient for the neuron we are interested in to 1 4. Backpropagate to get reconstructed image (gradient on the image) Springenberg, Dosovitskiy, et al. Striving for Simplicity: The All Convolutional Net. ICLR 2015 21
  22. 22. Gradient-based approach Springenberg, Dosovitskiy, et al. Striving for Simplicity: The All Convolutional Net. ICLR 2015 22
  23. 23. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 23
  24. 24. Optimization approach Simonyan et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, 2014 Obtain the image that maximizes a class score (or a neuron activation) 1. Forward random image 2. Set the gradient of the scores vector to be [0,0,0…,1,...,0,0] 3. Backprop to get gradient on the image 4. Update image (small step in the gradient direction) 5. Repeat 24
  25. 25. Optimization approach Simonyan et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, 2014 25
  26. 26. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 26
  27. 27. DeepDream https://github.com/google/deepdream 27
  28. 28. DeepDream 1. Forward image up to some layer (e.g. conv5) 2. Set the gradients to equal the layer activations 3. Backprop to get gradient on the image 4. Update image (small step in the gradient direction) 5. Repeat 28
  29. 29. DeepDream 1. Forward image up to some layer (e.g. conv5) 2. Set the gradients to equal the layer activations 3. Backprop to get gradient on the image 4. Update image (small step in the gradient direction) 5. Repeat At each iteration, the image is updated to boost all features that activated in that layer in the forward pass. 29
  30. 30. DeepDream 30 More examples here
  31. 31. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 32
  32. 32. Neural Style Style image Content image Result 33Gatys et al. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016
  33. 33. Neural Style Extract raw activations in all layers. These activations will represent the contents of the image. 34Gatys et al. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016
  34. 34. Neural Style V = ● Activations are also extracted from the style image for all layers. ● Instead of the raw activations, gram matrices (G) are computed at each layer to represent the style. E.g. at conv5 [13x13x256], reshape to: 169 256 ... G = VT V The Gram matrix G gives the correlations between filter responses. 35
  35. 35. Neural Style match content match style Match gram matrices from style image Match activations from content image 36Gatys et al. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016
  36. 36. Neural Style match content match style Match gram matrices from style image Match activations from content image 37Gatys et al. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016
  37. 37. Neural Style 38Gatys et al. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016
  38. 38. Neural Style 39 Improvements since Gatys et al.: Johnson et al. Perceptual Losses for Real-Time Style Transfer and Super-Resolution ECCV 2016 [code] ● 100x faster than original method ● Needs to train a network for a small amount of time for each style to render Luan et al. Deep Photo Style Transfer. arXiv Apr 2017 [code] ● Constraint input-output transformation to be locally affine in colorspace
  39. 39. 40 Content Image Style Image Result Luan et al. Deep Photo Style Transfer. arXiv Apr 2017
  40. 40. 41 Content Image Style Image Result Luan et al. Deep Photo Style Transfer. arXiv Apr 2017
  41. 41. Visualization ● Learned weights ● Activations from data ● Representation space ● Gradient-based ● Optimization-based ● DeepDream ● Neural Style 42
  42. 42. Questions ?

×