This document outlines various neural network architectures for solving different types of problems using deep learning. It discusses classification, regression, multimodal classification, video question answering, image question answering, and image-to-image translation problems. For each problem type, it provides examples of possible neural network architectures that could be used to map input data to output classifications or predictions. The goal is to help attendees understand how to design neural network architectures suited to their specific problems and mapping input to output.

1. Design
How to solve a problem using
Deep Learning
AI for Good Workshop
July 2019
Session 3
https://sites.google.com/view/AIforEveryone

2. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

3. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #1
Is it possible to map hand signs for deaf  speech
 Hand signs   Speech

4. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
A B
•

5. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #2
Is it possible to “see the depth” for visual impaired
 Yes / no ?

6. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz #2
Is it possible to feel the depth for visual impaired

7. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

8. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Topics in this presentation
•New way to think from today !
• How to use Deep Learning to solve any problem
• How to model & architect Neural Networks

9. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
New way to think from today !

10. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
New way to think from today !
Do you have
 Large dataset?
 Compute resource (can pay easy to cloud)
 Ability to use frameworks such as Tensorflow

11. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1
How to design an neural network architecture?

12. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1How to design an neural network architecture?
How to map x to y?
What
architecture
can connect
x to y?

13. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1How to design an neural network architecture?
Predicted
class
How to map x to y?
What architecture can map x to y?

14. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Deep Learning idea-a-thon:
How to solve a problem
by designing novel neural network architectures?
CNN
RNN
Fully
Connected
Merge
CNN is suitable architecture to extract
features out of any data that has
spatial relationships. For example, An
photo image spatial info..or info
spread across space
In general, a recurrent neural network could
be considered as the best neural network
model for extracting features out of
temporal data. A temporal data is basically
a data that varies over time such as ECG,
music, speech, sentences, words. LSTM is
a advanced RNN
To classify / predict
To combine the learnings of
two neural networks.
(to combine the power of 2
people’s brains)
Attention
To learn to focus on the
most important aspects to
achieve a particular task
Generative
Deep Learning
To model creative tasks as
to compose a music or
to come up a painting

15. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
How to use
Deep learning
to learn
to connect A and B

16. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learning objective:
How to learn to map A  B
Deep
Learning
A
B
Rules or math function
to find B given A.
B = f(A)
What should be f()?

17. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
How to translate A to B

18. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

19. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map Xy
• Examples of x, y

20. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map Xy

21. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map A  B

22. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone

23. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Fun activity

24. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learning objective:
How do solve a problem
using Deep Learning?
Case study: self driving car

25. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key Learning objective
for Day 1
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

26. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #1
classification problems
CNN
Fully
Connected
Prediction of what
of objects is in this
photo

27. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1
CNN
Fully
Connected
Turn car left
Turn car right
Don’t Turn the car

28. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Learn to map x->y

29. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1

30. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #1

31. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Camera view ? steering angle
•
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
Credits: Nvidia

32. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
what the neural network learns depends
upon its purpose
• How to set the purpose of a “learning machine”?

33. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
•

34. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
What is the function to map x->y
•

35. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #2
Regression problems
CNN
Fully
Connected
Continuous number

36. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems
CNN
Fully
Connected
Turn the car by 10 degrees

37. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems

38. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #2
Regression problems
•

39. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification
CNN
Fully
Connected
Predicted
class
RNN
Merge

40. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification

41. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification

42. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Neural Network Design Pattern #3
multimodal classification
•
Quiz: What was x and y in the problem?

43. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Neural Network Design Pattern #3
• Quiz: What was x and y in the problem?

44. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
multimodal classification

45. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Answering Visual Questions from Blind People

46. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #3
multimodal classification

47. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #3
multimodal classification
•

48. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #3

49. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz/Thinking activity
how to classify a genomic sequence
healthcare/genomic sequence classification

50. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
How to Solve this problem with deep learning:
how to classify a genomic sequence
?
?
?
Credits: DeepMotif (ICLR’16) http://www.cs.virginia.edu/yanjun/paperA14/2017_demo_slides.pdf
This is a classification problem.
A: Sequence of characters
B: Category

51. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
How to Solve this problem with deep learning:
how to classify a genomic sequence
?
?
?

52. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
how to classify a genomic sequence

53. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #4
video question answering
CNN
Fully
Connected
Predicted
class
RNN
Merge
Video
Question
RNN

54. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #4
video question answering

55. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering
•

56. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

57. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

58. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

59. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

60. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #4
video question answering

61. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #5
image question answering
CNN
Answer
(text)
RNN
Merge
Image
(photo)
Question
(text)
RNN

62. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #5
image question answering

63. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #6

64. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #6

65. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Architecture patterns
Neural Network Design Pattern #7
image to image translation
CNN
Image
(text)
Image
(photo)
CNN
Features of
the image

66. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

67. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

68. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

69. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Neural Network Design Pattern #7
image to image translation

70. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Can a neural network output 2 types
of output?
• How to architect Multi output network?

71. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
Multi output network

72. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
how to extract features from multiple
sources ?
• I have a input of audio and video and text. How to combine the features
from all 3?

73. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Quiz
how to extract features from multiple sources ?

74. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Our inspiration / acknowledgements
Friendly approaches :
1) KERAS.io
François Chollet’s
Book on “Deep Learning with Python”
2) Deeplearning.ai (Coursera.org)
Andrew Ng
3) Udacity Deep Learning Free course
4) Google Machine Learning Course
https://developers.google.com/machine-learning/crash-course/

75. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
More inspiration/ acknowledgements
Excellent Resources
• Stanford cs231 n
http://cs231n.stanford.edu
• MIT Deep Learning
http://introtodeeplearning.com/
https://deeplearning.mit.edu
• IIT Madras
my classes notes with Prof. Anurag (Deep Learning)

76. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
What did you learn?
(Day 1)
How to solve a problem with Deep Learning ?
How to design an neural network architecture?

77. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
You can map any thing to anything with Deep Learning AB
• Anything  Machine Learns to map  Anything

78. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
How to design a architectures to handle various types of A  B
How to design this architecture?
CNN / RNN ?
Fully Connected or RNN ?
Predicted
class
RNN /
Fully connected ?
Merge / Attention ?
Video
Question
A  --------------------------------------------------------------------------------------------------------------------------  B

79. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
From architecture to code ? (just learn early glimpse of how code looks like.)
Just 1 page of code is enough for a complex problems such as video question answering

80. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Key ideas to remember
If you didn’t understand the code*, just don’t worry.
• Why not to worry?
• Solving the video question answering problem was not possible until 2017.
• To solve this problem, it required dozens of experts in Google lab to dedicate 6 months of effort!
*code = code in the previous slide

81. Acknowledgments & Credits are mentioned to inspirational resources presented in the end of presentation. For more like this, https://sites.google.com/view/AIforEveryone
Team activity
idea-a-ton
Goal of this thinking activity
How to solve a problem with Deep Learning ?
How to design an neural network architecture?