1. Deep learning
Made Easy with Deep
Features
Piotr Teterwak
Dato, Machine Learning Engineer

2. 2
Hello, my name is…
Piotr Teterwak
Machine Learning Engineer,
Dato

3. 3
Who is Dato?

4. Graphlab Create: Production ML Pipeline
DATA
YourWebServiceor
IntelligentApp
ML
Algorithm
Data
cleaning
&
feature
eng
Offline
eval &
Parameter
search
Deploy
model
Data engineering Data intelligence Deployment
Goal: Platform to help implement, manage, optimize entire pipeline

5. Deep Learning

6. Today’s talk

7. Today’s talk
Features in ML
Deep Neural
Networks learn
features
Using those
learned features
for new tasks
Productionizing

8. Features are key to machine learning

9. 9
Simple example: Spam filtering
• A user opens an email…
- Will she thinks its spam?
• What’s the probability email is spam?
Text of email
User info
Source info
Input: x
MODEL
Yes!
No
Output:
Probability of y

10. 10
Feature engineering:
the painful black art of transforming raw inputs
into useful inputs for ML algorithm
• E.g., important words, complex transformation of input,…
MODEL
Yes!
No
Output:
Probability of y
Feature
extraction
Features: Φ(x)
Text of email
User info
Source info
Input: x

11. Deep Learning for Learning Features

12. 12
Linear classifiers
• Most common classifier
- Logistic regression
- SVMs
- …
• Decision correspond to
hyperplane:
- Line in high dimensional
space
w0 + w1 x1 + w2 x2 > 0 w0 + w1 x1 + w2 x2 < 0

13. 13
What can a simple linear classifier represent?
AND
0
0
1
1

14. 14
What can a simple linear classifier represent?
OR
0
0
1
1

15. 15
What can’t a simple linear classifier represent?
XOR
0
0
1
1
Need non-linear features

16. 16
Non-linear feature embedding
0
0
1
1

17. 17
Graph representation of classifier:
Useful for defining neural networks
x
1
x
2
x
d
y
…
1
w2 w0 + w1 x1 + w2 x2 + … + wd xd
> 0, output 1
< 0, output 0
Input Output

18. 18
What can a linear classifier represent?
x1 OR x2 x1 AND x2
x
1
x
2
1
y
-0.5
1
1
x
1
x
2
1
y
-1.5
1
1

19. Solving the XOR problem: Adding a layer
XOR = x1 AND NOT x2 OR NOT x1 AND x2
z
1
-0.5
1
-1
z1 z2
z
2
-0.5
-1
1
x
1
x
2
1
y
1 -0.5
1
1
Thresholded to 0 or 1

20. 20
http://deeplearning.stanford.edu/wiki/images/4/40/Network3322.png
Deep Neural Networks
P(cat|x)
P(dog|x)

21. 21
Deep Neural Networks
• Can model any function with enough hidden units.
• This is tremendously powerful: given enough units, it is
possible to train a neural network to solve arbitrarily
difficult problems.
• But also very difficult to train, too many parameters
means too much memory+computation time.

22. 22
Neural Nets and GPU’s
• Many operations in Neural Net training can happen in
parallel
• Reduces to matrix operations, many of which can be
easily parallelized on a GPU.

23. 24
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

24. 25
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

25. 26
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

26. 27
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

27. 28
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

28. 29
Convolutional Neural Nets
• Strategic removal of edges
Input Layer
Hidden Layer

29. 30
Convolutional Neural Nets
http://ufldl.stanford.edu/wiki/images/6/6c/Convolution_schematic.gif

30. 31
Pooling layer
Ranzato, LSVR tutorial @ CVPR, 2014. www.cs.toronto.edu/~ranzato

31. 32
Pooling layer
http://ufldl.stanford.edu/wiki/images/6/6c/Pooling_schematic.gif

32. 33
Final Network
Krizhevsky et al.
‘12

33. Applications to computer vision

34. 35
Image features
• Features = local detectors
- Combined to make prediction
- (in reality, features are more low-level)
Face!
Eye
Eye
Nose
Mouth

35. 36
Standard image classification approach
Input
Computer$vision$features$
SIFT$ Spin$image$
HoG$ RIFT$
Textons$ GLOH$
Slide$Credit:$Honglak$Lee$
Extract features Use simple classifier
e.g., logistic regression, SVMs
Face

36. 37
Many hand crafted features exist…
Computer$vision$features$
SIFT$ Spin$image$
HoG$ RIFT$
Textons$ GLOH$
Slide$Credit:$Honglak$Lee$
… but very painful to design

37. 38
Change image classification approach?
Input
Computer$vision$features$
SIFT$ Spin$image$
HoG$ RIFT$
Textons$ GLOH$
Slide$Credit:$Honglak$Lee$
Extract features Use simple classifier
e.g., logistic regression, SVMs
FaceCan we learn features
from data?

38. 39
Use neural network to learn features
Input
Learned hierarchy
Output
Lee et al. ‘Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations’ ICML 2009

39. Sample results
• Traffic sign recognition
(GTSRB)
- 99.2% accuracy
• House number recognition
(Google)
- 94.3% accuracy
40

40. Krizhevsky et al. ’12:
60M parameters, won 2012 ImageNet competition
41

41. 42
ImageNet 2012 competition: 1.2M images, 1000 categories
42

42. 43
Application to scene parsing
©Carlos Guestrin 2005-2014
Y LeCun
MA Ranzato
Semantic Labeling:
Labeling every pixel with the object it belongs to
[ Farabet et al. ICML 2012, PAMI 2013]
Would help identify obstacles, targets, landing sites, dangerous areas
Would help line up depth map with edge maps

43. A quick demo!
44

44. Challenges of deep learning

45. Deep learning score card
Pros
• Enables learning of features rather
than hand tuning
• Impressive performance gains on
- Computer vision
- Speech recognition
- Some text analysis
• Potential for much more impact
Cons

46. Deep learning workflow
Lots of
labeled data
Training set
Validation set
80%
20%
Learn deep
neural net
model
Validate

47. Deep learning score card
Pros
• Enables learning of features rather
than hand tuning
• Impressive performance gains on
- Computer vision
- Speech recognition
- Some text analysis
• Potential for much more impact
Cons
• Computationally really expensive
• Requires a lot of data for high
accuracy
• Extremely hard to tune
- Choice of architecture
- Parameter types
- Hyperparameters
- Learning algorithm
- …
• Computational + so many choices =
incredibly hard to tune

48. 49
Can we do better?
Input
Learned hierarchy
Output
Lee et al. ‘Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations’ ICML 2009

49. Deep features:
Deep learning
+
Transfer learning

50. 51
Transfer learning:
Use data from one domain to help learn on another
Lots of data:
Learn
neural net
Great
accuracy
Some data: Neural net as
feature extractor
+
Simple classifier
Great accuracy on
new problem
Old idea, explored for deep learning by Donahue et al. ’14

51. 52
What’s learned in a neural net
Neural net trained for Task 1
Very specific to Task 1More generic
Can be used as feature extractor
vs.

52. 53
Transfer learning in more detail…
Neural net trained for Task 1
Very specific to Task 1More generic
Can be used as feature extractor
Keep weights fixed!
For Task 2, learn only end part
Use simple classifier
e.g., logistic regression, SVMs
Class?

53. 54
Using ImageNet-trained network as extractor for
general features
• Using classic AlexNet architechture pioneered by Alex Krizhevsky
et. al in ImageNet Classification with Deep Convolutional Neural
Networks
• It turns out that a neural network trained on ~1 million images of
about 1000 classes makes a surprisingly general feature extractor
• First illustrated by Donahue et al in DeCAF: A Deep Convolutional
Activation Feature for Generic Visual Recognition
54

54. 55
Caltech-101
55

55. Transfer learning with deep features
Training set
Validation set
80%
20%
Learn
simple
model
Some
labeled data
Extract
features with
neural net
trained on
different task
Validate
Deploy in
production

56. Demo
58

57. What else can we do with Deep Features?
59

58. Finding similar images
60

59. Applications to text data

60. Simple text classification with bag of words
aardvark 0
about 2
all 2
Africa 1
apple 0
anxious 0
...
gas 1
...
oil 1
…
Zaire 0
Use simple classifier
e.g., logistic regression, SVMs
Class
?
One “feature” per word

61. Word2Vec: Neural network for finding word
representation Mikolov et al. ‘13
Skip-gram Model: From a word, predict nearby words in sentence
dog
A went for a walk
Neural net
Viewed as deep
features

62. Word2Vec: Neural network for finding high
dimensional representation per word Mikolov et al. ‘13
http://www.folgertkarsdorp.nl/word2vec-an-introduction/

63. 65
Related words placed nearby high dim space
Projecting 300 dim space into 2 dim with PCA (Mikolov et al. ’13)

64. Blog corpus
Haha
Yea
Hahaha
Hahah
Lisxc
Umm
Hehe
laughingoutloud
LOL
Closest words
in 300 dim
Predicts gender of author with 79% accuracy

65. ML in production
(Or how this is relevant to data scientists)

66. 2015: Production ML pipeline
DATA
YourWebServiceor
IntelligentApp
ML
Algorithm
Data
cleaning
&
feature
eng
Offline
eval &
Parameter
search
Deploy
model
Data engineering Data intelligence Deployment
Using deep learning
Goal: Platform to help implement, manage, optimize entire pipeline

67. In real life

68. Take Home Message

69. 71
Take Home Message
Use simple classifier
e.g., logistic regression, SVMs
Class?
Deep Features are remarkable!

70. 72
CONF.DATO.COM

71. 73
Dato Office Hours @ Galvanize SF
• Bring your laptop & some data & we’ll help you get started
• When: Thurs (tomorrow) 2:30p-5p followed by beers
• Where: Galvanize – 44 Tehama St. (SOMA) in SF
• Talk to me/email me: piotr@dato.com
+

72. Get the software: dato.com/download
Learn: dato.com/learn
Learn more: blog.dato.com
Join us: we’re hiring lots!
Contact me: piotr@dato.com

73. 75
Go create something! [with Dato]
Data
Engineering
Data
Intelligence
Deployment
• Fast & scalable
• Rich data type support
• Visualization
• App-oriented ML
• Supporting utils
• Extensibility
• Batch & always-on
• RESTful interface
• Elastic & robust