The document outlines a presentation on solving problems using deep learning. It discusses key learning objectives of how to solve problems with deep learning and design neural network architectures. Various neural network design patterns are presented for different types of problems, such as classification, regression, multimodal classification, video question answering, and image translation. The presentation emphasizes that deep learning can be used to map any input to any output.

1. Design
How to solve a problem using
Deep Learning
AI for Good Workshop
July 2019
Session 3
https://sites.google.com/view/AIforEveryone

2. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

3. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #1
Is it possible to map hand signs for deaf  speech
 Hand signs   Speech

4. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
A B
•

5. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #2
Is it possible to “see the depth” for visual impaired
 Yes / no ?

6. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #2
Is it possible to feel the depth for visual impaired

7. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

8. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Topics in this presentation
•New way to think from today !
• How to use Deep Learning to solve any problem
• How to model & architect Neural Networks

9. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
New way to think from today !

10. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
New way to think from today !

11. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Deep Learning idea-a-thon:
How to solve a problem
by designing novel neural network architectures?
CNN
RNN
Fully
Connected
Merge
CNN is suitable architecture to extract
features out of any data that has
spatial relationships. For example, An
photo image spatial info..or info
spread across space
In general, a recurrent neural network could
be considered as the best neural network
model for extracting features out of
temporal data. A temporal data is basically
a data that varies over time such as ECG,
music, speech, sentences, words. LSTM is
a advanced RNN
To classify / predict
To combine the learnings of
two neural networks.
(to combine the power of 2
people’s brains)
Attention
To learn to focus on the
most important aspects to
achieve a particular task
Generative
Deep Learning
To model creative tasks as
to compose a music or
to come up a painting

12. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Our inspiration / acknowledgements
Friendly approaches :
1) KERAS.io
François Chollet’s
Book on “Deep Learning with Python”
2) Deeplearning.ai (Coursera.org)
Andrew Ng
3) Udacity Deep Learning Free course
4) Google Machine Learning Course
https://developers.google.com/machine-learning/crash-course/

13. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
More inspiration/ acknowledgements
Excellent Resources
• Stanford cs231 n
http://cs231n.stanford.edu
• MIT Deep Learning
http://introtodeeplearning.com/
https://deeplearning.mit.edu
• IIT Madras
my classes notes with Prof. Anurag (Deep Learning)

14. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Message: Using
Representation learning to
learn to represent A as B

15. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learning objective:
How did the magic happen?
Deep
Learning

16. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learning objective:
How did the magic happen?

17. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

18. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map Xy
• Examples of x, y

19. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
A  B

20. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

21. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

22. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

23. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Fun activity

24. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learning objective:
How do solve a problem
using Deep Learning?
Case study: self driving car

25. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Camera view ? steering angle
•
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
Credits: Nvidia

26. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
•

27. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map x->y

28. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
What is the function to map x->y
•

29. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

30. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #1
classification problems
CNN
Fully
Connected
Prediction of what
of objects is in this
photo

31. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1
CNN
Fully
Connected
Turn car left
Turn car right
Don’t Turn the car

32. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1

33. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1

34. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #2
Regression problems
CNN
Fully
Connected
Continuous number

35. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems
CNN
Fully
Connected
Turn the car by 10 degrees

36. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems

37. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems
•

38. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification
CNN
Fully
Connected
Predicted
class
RNN
Merge

39. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification

40. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification

41. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Neural Network Design Pattern #3
multimodal classification
•
Quiz: What was x and y in the problem?

42. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Neural Network Design Pattern #3
• Quiz: What was x and y in the problem?

43. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
multimodal classification

44. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Answering Visual Questions from Blind People

45. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #3
multimodal classification

46. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification
•

47. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #3

48. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #4
video question answering
CNN
Fully
Connected
Predicted
class
RNN
Merge
Video
Question

49. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #4
video question answering

50. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering
•

51. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

52. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

53. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

54. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

55. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

56. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #5
image question answering
CNN
Answer
(text)
RNN
Merge
Image
(photo)
Question
(text)
RNN

57. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #5
image question answering

58. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #6

59. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #6

60. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #7
image to image translation
CNN
Image
(text)
Image
(photo)
CNN
Features of
the image

61. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

62. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

63. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

64. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

65. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Can a neural network output 2 types
of output?
• How to architect Multi output network?

66. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Multi output network

67. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
how to extract features from multiple
sources ?
• I have a input of audio and video and text. How to combine the features
from all 3?

68. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
how to extract features from multiple sources ?

69. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
What did you learn?
(Day 1)
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

70. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
You can map any thing to anything with Deep Learning AB
• Anything  Machine Learns to map  Anything

71. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
How to design a architectures to handle various types of A  B
How to design this architecture?
CNN / RNN ?
Fully Connected or RNN ?
Predicted
class
RNN /
Fully connected ?
Merge / Attention ?
Video
Question
A  --------------------------------------------------------------------------------------------------------------------------  B

72. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
From architecture to code ? (just learn early glimpse of how code looks like.)
Just 1 page of code is enough for a complex problems such as video question answering

73. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
If you didn’t understand the code*, just don’t worry.
• Why not to worry?
• Solving the video question answering problem was not possible until 2017.
• To solve this problem, it required dozens of experts in Google lab to dedicate 6 months of effort!
*code = code in the previous slide

74. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Team activity
idea-a-ton
Goal of this thinking activity
How to solve a problem with Deep Learning ?
How to design an neural network architecture?