This document provides an agenda and introduction for a deep learning workshop. It includes an introduction to the instructor, ground rules for the workshop, and a tentative agenda that will cover topics like neural networks, convolutional neural networks, and RNNs. Hands-on exercises are planned for each section. The workshop will be held in a Jupyter notebook environment on Google Colab, allowing participants to run Python code directly in their browser.

1. GOOD
EVENING!

2. Polyglot Ninja
Memphis, TN area
Co-Directory of MemPy
Data Science, Machine Learning, Mobile Apps
Wannabe Game Designer
Pluralsight Author
@poweredbyaltnet
Douglas Starnes
@poweredbyaltnet

3. Ground Rules
• I’m going to throw a lot of information at you
• We have 4 hours together (that’s 240 minutes)
• There are 3 breaks for a total of 30 minutes
• There are 89 slides (~ 45 min)
• There are 10 exercises (~17 minutes each)
• There will be questions
• There will be problems logging in
• Google could go on vacation
• But the wifi will rock!
• The agenda is a guide.
• I’m here to focus on what you are interested in.
• If I ask a question, it’s OK to answer.
• Like .. out loud
• I honestly welcome your feedback.
• I’ve made changes to this workshop based on comments (both good and bad) from
my last workshop two weeks ago. @poweredbyaltnet

4. “AGENDA”
(verrrrrrrry tentative, take with a grain of salt, or 10-20 grains)
• Intro (5 mins) – SLIDES
• Jupyter notebook / Google Colab (10 mins) – SLIDES
• Python primer (20 mins) – Hands on
• PyData (10 mins) – hands on
• Break (10 mins)
• Neural Network Concepts (30 mins) – slides
• Neural Network Classification (15 mins) – hands on
• Neural Network Regression (15 mins) – hands on
• Break (10 mins)
• Convolutional Neural Network Concepts (20 mins) - slides
• Convolutional Neural Network MNIST (20 mins) – hands on
• Convolutional Neural Network CIFAR-10 (15 mins) – hands on
• Break (10 mins)
• RNN Concepts (10 mins) – slides
• RNN IMDB (10 mins) – hands on
• Tensorboard – and other cool stuff? (25 mins) – hands on
@poweredbyaltnet

5. MEET GOOGLE
COLAB
http://colab.research.google.com
@poweredbyaltnet

6. This is a notebook
@poweredbyaltnet

7. Python code goes into cells
@poweredbyaltnet

8. Shift-Enter executes a cell
@poweredbyaltnet

9. It’s like a Python session in a webpage
@poweredbyaltnet

10. Actually all the work is done in Google Cloud
@poweredbyaltnet

11. Syntax highlighting and code completion
@poweredbyaltnet

12. Shell commands
@poweredbyaltnet

13. Rich text using Markdown
@poweredbyaltnet

14. Everything is stored in Google Drive
@poweredbyaltnet

15. You even get access to GPUs and TPUs!
@poweredbyaltnet

16. You can get all of this
and more for only
$19.99!
@poweredbyaltnet

17. Actually it’s free
@poweredbyaltnet

18. The only price of
admission is to login
with a GMail address
@poweredbyaltnet

19. So let’s get started
Welcome to
Deep Learning for Folks Without a Ph.D.^
@poweredbyaltnet

20. http://bit.ly/dl-woaphd
@poweredbyaltnet

21. @poweredbyaltnet
Artificial Intelligence
Machine Learning
Deep Learning
D
a
t
a
S
c
i
e
n
c
e

22. Classification
• Cats
• have 4 legs
• have fur
• have long tails
• have short noses
• have small upright ears
• Dogs
• have 4 legs
• have fur
• have long tails
• have long noses
• have large floppy ears
• Pigs
• have 4 legs
• have no/little fur
• have short tails
• have short noses
• have large upright ears
@poweredbyaltnet

23. Dog
• Four legs
• Fur
• Long nose
@poweredbyaltnet

24. Pig
• Four legs
• Little fur
• Large upright ears
@poweredbyaltnet

25. Cat
• Four legs
• Fur
• Small upright ears
@poweredbyaltnet

26. Dog
• Four legs
• Furry
• Long nose
@poweredbyaltnet

27. @poweredbyaltnet

28. @poweredbyaltnet
Neural networks are loosely inspired by how neuroscientists today think the brain might possibly work.
(assuming they are on the right track)

29. @poweredbyaltnet
node
layer
input
output
hidden

30. @poweredbyaltnet
A neural network with more than two hidden layers is called a deep neural network.

31. @poweredbyaltnet
Sepal Length Sepal Width Petal Length Petal Width Species
5.1 3.5 1.4 0.2 setosa
… … … … …
7.0 3.2 4.7 1.4 versicolor
… … … … …
5.7 2.8 4.1 1.3 virginica
rows -> observations
columns -> features
feature vector / independent variables -> observed values
target / dependent variable(s) -> what we want to predict
feature vector target
train a model that, given a feature vector, will (hopefully accurately) assign a target
train a model that, given values for Sepal Length, Sepal Width, Petal Length, Petal Width, will
assign one value from the set of setosa, versicolor, or virginica

32. @poweredbyaltnet
input
5.1
3.5
1.4
0.2
outputhidden
.55
.30
.15
55%
setosa
30%
versicolor
15%
virginica

33. @poweredbyaltnet
Is this
prediction
correct?

34. @poweredbyaltnet
There is only one trick question in
the entire talk. This is it.

35. @poweredbyaltnet
During training, we are not
concerned with making a correct
prediction, but rather making a
prediction that is as close to
correct as possible.

36. @poweredbyaltnet

37. @poweredbyaltnet
Prediction
.55
.30
.15
Actual
1.0
0
0
setosa
versicolor
virginica

38. @poweredbyaltnet
I think we
can do
better!

39. @poweredbyaltnet
Prediction
.55
.30
.15
Actual
1.0
0
0
.9999999
.0000001
.0000001
More 9’s please!

40. @poweredbyaltnet
Prediction Actual
Cost function
Model Dataset
Optimizer

41. @poweredbyaltnet
learning rate
If the learning rate is too large, we will skip over the minimum.
If the learning rate is too small, we will waste time.

42. @poweredbyaltnet
Prediction Actual
Cost function
Model Dataset
Optimizerbackpropagation

43. @poweredbyaltnet
The model is trained
Now what?

44. @poweredbyaltnet
DatasetDataset
Training Dataset
Test Dataset
70-80%
20-30%
Model
feature vector
Setosa Versicolor Virginica
.99 .002 .008
.98 .015 .005
.002 .99 .008
.05 .04 .991
Setosa Versicolor Virginica
1.0 0 0
1.0 0 0
1.0 0 0
0 0 1.0
t
a
r
g
e
t
prediction
𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠
𝑡𝑜𝑡𝑎𝑙 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠

46. @poweredbyaltnet
playground.tensorflow.org

47. @poweredbyaltnet

48. @poweredbyaltnet

49. @poweredbyaltnet
Meet MNIST
Scans of handwritten digits (0-9)
Each image is 28x28 pixels
Feature vector is the entire
image (784 pixels)
The target is the digit
represented (0-9)
Each pixel value is 0 (black)
to 255 (white)
Interesting variations
(ie. fashion MNIST)
Each pixel (for now) is a
feature

50. @poweredbyaltnet
7
8
4
input
(implied)
1
2
8
hidden
r
e
l
u

51. Why Activation Layers
• Without activation layers, we can only approximate linear functions
(e.g. linear regression)
• Neural networks are universal function approximators
• Activation layers introduce non-linearity
• Non-linear functions are more complex
• The real world is often very complex

52. Linear
𝑓 𝑥 = 𝑥
@poweredbyaltnet

53. ReLU (Rectified Linear Unit)
𝑓 𝑥 =
0, 𝑥 < 0
𝑥, 𝑥 ≥ 0
@poweredbyaltnet

54. Sigmoid
𝑓 𝑥 =
1
1 + 𝑒−𝑥
@poweredbyaltnet

55. Hyperbolic Tangent
𝑓 𝑥 = tanh(𝑥)
@poweredbyaltnet

56. @poweredbyaltnet
7
8
4
input
(implied)
1
2
8
hidden
r
e
l
u
1
0
output
s
o
f
t
m
a
x

57. @poweredbyaltnet
from math import exp
i = [100, 101, 102]
def softmax(x):
return exp(i[x]) / sum([exp(j) for j in i])
print([softmax(a) for a in range(len(i))])
[0.09003057317038046, 0.24472847105479764, 0.6652409557748219]
9% 24% 67%
𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑖 𝑥 =
𝑒 𝑖 𝑥
𝑗=1
𝑘
𝑒 𝑖 𝑗
𝑓𝑎𝑘𝑒_𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑖 𝑥 =
𝑖 𝑥
𝑗=1
𝑘
𝑖𝑗
Exacerbate
the error
[0.33003300330033003, 0.3333333333333333, 0.33663366336633666]
33% 33% 34%

58. @poweredbyaltnet
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(60000, 784).astype(‘float32’) / 255 # also apply to X_test
Y_train = np_utils.to_categorical(y_train, 10) # one hot encoding, also apply to y_test
model = Sequential()
model.add(Dense(128, input_shape=(784,))) # hidden layer, input implied
model.add(Activation(‘relu’))
model.compile(loss=‘categorical_crossentropy’, optimizer=SGD(), metrics=[‘accuracy’])
model.fit(X_train, Y_train, batch_size=128, epochs=20) # around 94%
score = model.evaluate(X_test, Y_test)
model.add(Dense(10)) # output layer
model.add(Activation(‘softmax’))

59. @poweredbyaltnet
That’s all there is to
it?
Surely there is a
catch.

60. @poweredbyaltnet
28px
28px
1
0 7,840
101,77028px
28px
1
2
8
1
0
480px
640px
>300,000

61. @poweredbyaltnet

62. @poweredbyaltnet
overfitting

63. @poweredbyaltnet

64. @poweredbyaltnet
Goal: consider fewer, more relevant, features
How to determine which features are relevant?
Let the network decide!
Convolutional Neural Network (CNN)
Three new layers
Convolutional layers – detect features
Pooling layers – summarize features
Features are then used in dense layers
Dropout – prevent overfitting

65. @poweredbyaltnet
i
n
p
u
t
C
o
n
v
2
D

66. @poweredbyaltnet
1 0 0
1 1 1
0 1 1
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
filter/kernel

67. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6

68. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6 4

69. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6 4 3

70. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6 4 3 4

71. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6 4 3 4
5

72. @poweredbyaltnet
1 0 0 1
1 1 1 0
0 1 1 1
1 1 1 0
1 0
1 1
0 1
1 1
0 0 1 1
0 1 1 0
0 1
0 0
1 0 0
1 1 1
0 1 1
6 4 3 4
5 5 4 4
4 5 4 4
4 4 3 2
convolution

73. @poweredbyaltnet
i
n
p
u
t
C
o
n
v
2
D
C
o
n
v
2
D
M
a
x
P
o
o
l
i
n
g
2
D
r
e
l
u

74. @poweredbyaltnet
6 4 3 4
5 5 4 4
4 5 4 4
4 4 3 2
6

75. @poweredbyaltnet
6 4 3 4
5 5 4 4
4 5 4 4
4 4 3 2
6 4

76. @poweredbyaltnet
6 4 3 4
5 5 4 4
4 5 4 4
4 4 3 2
6 4
5

77. @poweredbyaltnet
6 4 3 4
5 5 4 4
4 5 4 4
4 4 3 2
6 4
5 4

78. @poweredbyaltnet
i
n
p
u
t
C
o
n
v
2
D
C
o
n
v
2
D
M
a
x
P
o
o
l
i
n
g
2
D
r
e
l
u
D
r
o
p
o
u
t

79. @poweredbyaltnet
... ... ... ... ... ...

80. @poweredbyaltnet
i
n
p
u
t
C
o
n
v
2
D
C
o
n
v
2
D
M
a
x
P
o
o
l
i
n
g
2
D
r
e
l
u
D
r
o
p
o
u
t
F
l
a
t
t
e
n
s
o
f
t
m
a
x
D
e
n
s
e
r
e
l
u
D
r
o
p
o
u
t

81. @poweredbyaltnet
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import Flatten, Conv2D, MaxPooling2D
(X_train, y_train), (X_valid, y_valid) = mnist.load_data()
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(n_classes, activation='softmax'))

82. and _____?
butter and _________bread / garlic / herbs
peanut butter and _________jelly / bananas
@poweredbyaltnet

83. Recurrent Neural Network Cell
@poweredbyaltnet

84. t
tanh
t-1 t+1
xt
ht
@poweredbyaltnet

85. the quick brown fox jumped over the lazy dog
x1 x2 x3 x4 x5 x6 x7 x8 x9
h1 h2 h3 h4 h5 h6 h7 h8 h9
@poweredbyaltnet

86. from keras.layers import SimpleRNN
...
model.add(SimpleRNN(64))
@poweredbyaltnet

87. Embedding Layers
• Map words into vectors
• Used as input to the RNN
• model = Sequential()
model.add(Embedding(2000, 16, input_length=200))
model.add(SimpleRNN(50))
model.add(Dense(1, activation=‘sigmoid’))
model.compile(loss=‘binary_crossentropy’, ...)
@poweredbyaltnet

88. Long Short Term Memory Cell
@poweredbyaltnet

89. 𝜎
x
x +
tanh𝜎
x
𝜎
tanh
xt ht
@poweredbyaltnet

90. from keras.layers import LSTM
...
model.add(LSTM(64))
@poweredbyaltnet

91. THANK YOU!
douglas@douglasstarnes.com
@poweredbyaltnet
http://douglasstarnes.com
(returning soon)