The document outlines the author's early experiences in machine learning, specifically focusing on deep learning techniques such as CNNs and RNNs. It discusses the advantages and flaws of deep neural networks, including their ability to learn high-level features and the challenges they face like adversarial examples. Additionally, it covers the modularity of DNNs and their applications beyond traditional machine learning tasks, alongside a brief overview of the hardware used for DNN computations.

1. From Conventional Machine
Learning to Deep Learning
and Beyond
by GB Chun Hao, Chang
Email: ccha97u@gmail.com

2. Who am I ?
I graduated from IDEA lab 4 months ago with a Master degree.
As a junior Machine Learning Engineer, my job is to understand the latest Deep
Learning research on Computer Vision and implement/modify them for the
Company’s needs
The following contents are my extremely limited experiences
(less than 4 months) and summary.
Please correct me if you found any error ;)

3. Overview
1. Brief introduction of Deep Learning
2. Two Popular Networks: CNN and RNN
3. Conventional Learning Model vs Deep Learning
4. The Strength of DNN
5. The Flaws of DNN
6. Beyond Regular Machine Learning Tasks
7. Appendix: Hardwares for DNN

4. What is Deep Learning
Deep Learning refers to Deep Neural Network.
It belongs to Neural Networks family, a branch of Machine Learning

5. Two popular branches in DNN
Convolutional
Neural Network
For Computer Vision
Tasks
Recurrent
Neural Network
For Sequential Problem

6. Convolutional Neural Network
The most popular Neural Network in Computer Vision
CNN is able to capture different levels of feature representation
The following pictures are generated purely by the weights of learned models
Image of the Bird Saxophone

7. CNN in Typical Computer Vision Tasks
CNN outperforms hand-crafted methods on Image
Classification and Object Detection tasks

8. Recurrent Neural Network
Recurrent Neural Network is able to memorize and recall the memory.
RNN are suitable for sequential data:
RNN is good at Time Series and Natural Language
Processing. RNN is Turing Complete

9. RNN Example - Word Embeddings
Mapping words into continuous vectors
(Some word embedding techniques does not need DNN at all. For example Word2Vec)

10. RNN Example - Language Translation

11. Shallow Neural Network
Shallow Neural Network is similar to most of
the conventional supervised models
Pros:
1.Easy to train and test
2.Able to approach any continuous function
Cons:
1.Performance depends on well-designed
features
2.Difficult to generalize the prediction

12. Deep Neural Network
Pros:
1.Automatically learn the High-level features representation
2.Modularity: DNN can be composed like LEGO bricks
3.Able to do Transfer Learning
Cons:
1.Requires tons of data for training
2.Expensive computation power
for training and testing (no CV)

13. What makes DNN so popular?
It has the three advantages:
1. Self-learned high-level features representations
2. Modularity
3. Transfer Learning
Why are these so important?

14. Self-Learned High-Level Feature Representation (⅓)
It means that you no longer need hand-crafted "Feature Engineering"
In the past, it takes several experts years to do the Feature Engineering on a
specific task

15. Self-Learned High-Level Feature Representation(⅔)
DNN simplify the pipelines of Machine Learning tasks (by removing Feature
Engineering), therefore researcher can use similar tools to solve different kinds of
tasks.
You can address different kinds of problems with similar
DNN models!

16. Self-Learned High-Level Feature Representation(3/3)
Example: The following LRCN model (CNN + RNN) is able to handle three
different kinds of problem: Activity Recognition, Image Description and Video Description

17. Modularity (¼)
DNN models can be composed just like building LEGO buildings

18. Modularity (2/4) :
For example, we want a DNN model for Object Detection, but we only have a
Image Classification DNN model(VGG-16).
We can either construct a complicated pipeline with conventional detectors or
compose a bigger DNN model with other DNN modules.
VGG-16
For Image Classification only
Object Detection Task
Modification

19. Modularity (¾) : From Image Classification to Object
detection
VGG-16
VGG-16
VGG-16 + ROI Polling Layer
VGG-16 + Region Proposal Network+ ROI Polling Layer
SVM
Selective Search
Selective Search
R-CNN
Fast R-CNN
Faster R-CNN
For Image Classification only
Complicated and slow
End-to-End and Fast

20. Modularity (4/4)
It’s also simple to replace one component in DNN
For example, ResNet-52 outperforms VGG-16 on Image Classification, we can
replace the VGG-16 with ResNet-52 in Faster R-CNN in order to improve the overall
performance in Object Detection
VGG-16 + Region Proposal Network+ ROI Polling Layer
Faster R-CNN
ResNet-52 + Region Proposal Network+ ROI Polling Layer
Faster R-CNN
VGG-16
ResNet-52
Replace

21. The learned knowledge of one task can be used in another task.
An apple detection model or data can help you to do orange detection!
Transfer Learning(¼)

22. Transfer Learning(2/4)
Conventional Machine Learning models are difficult to apply Transfer Learning.
The model depends on domain specific features and is sensitive to the data
distribution. Furthermore, different kinds of model(SVM, decision trees) can not
share their weights easily
Difficult

23. Transfer Learning(¾)
Use a pre-trained model and fine-tune it to adapt new domain tasks is a
common practice in DNN.
Example:
VGG-16
ImageNet Model
(Classification)
Style Recognition
Pedestrian Detection
Fine-Tuning
Fine-Tuning

24. Transfer Learning(4/4): Wisdom of the DNN Models
Many researchers not only publish their
thesis and source code. The Trained
Models are also shared on-line.
You can download the Pre-trained
Models, combine them or fine-tune them.
Caffe Model Zoo

25. DNN Minimize the gaps between areas
Text Data
Voice and
other Signals
Image and
Videos
Reinforcement
Learning
DNN can help you get into other research areas easier!

26. Are DNN models really that magical?
Hehehe, NO
DNN models are nice tools but not that
magical.
Let me give you some clues and
my experience

27. Flaws of DNN (⅓): MS-CNN Compromise on the Data Distribution
Do you find anything conflicting with the advantages of DNN models?

28. Flaws of DNN (⅔): TSN Compromise on Self-Learning Feature
Do you find anything conflicting with the advantages of DNN models?

29. Flaws of DNN(3/3): Adversarial Examples
Maybe it’s not a big deal, but Adversarial Examples somehow breaks the
belief of “DNN models are similar to human brains”

30. DNN are still wonderful tools to use
The above problems are not
serious and have solutions.
Furthermore, there are tons
of researchers are working
on better and smarter DNN
models.

31. Beyond Regular Machine Learning Tasks
Except for the regular classification and regression tasks. DNN models more than
that.
Once you can define and evaluate the task mathematically, you can apply DNN
models
Think DNN models as
“Trainable Program that does everything”

32. CNN model - Google Deep Dream

33. CNN model - Neural Style

34. CNN models - Generative Adversarial Network

35. RNN - Generated Kanji

36. RNN - Generated Latex Files

37. RNN - Composing Music

38. Combine CNN with RNN
Image + Time Series = Videos
and varieties of applications!

39. CNN+RNN: LRCN

40. CNN+RNN: Image Captioning
Demonstration Attention
Mechanism to sequentialize
an image by focusing on
different spots
RNN module decide which
part of image to focus

41. CNN+RNN: Image Based Question Answering

42. Deep Q-Networks (Reinforcement Learning)
DNN that implements Q-Learning algorithms

43. Neural Turing Machine
A Turing machine that is differentiable and can be trained on gradient descent
A DNN model that able to
control external storage

44. Memory Network End-to-End

45. Zero-Shot & One-Shot Learning
Models that depends on small dataset and able to detects never-seen classes
Researcher are using DNN to do Zero-Shot & One-Shot Learning tasks

46. Any Question? :)

47. Hardwares: Nvidia - Biggest Player in DNN
For normal users, the only choice to run DNN is Nvidia video cards
Nvidia will be focusing more on Intelligence Computations and Self Driving Cars in
future

48. Hardwares: Google Tensor Processing Unit
This Google homemade
TPU are mysterious and
not accessible for
outsiders.

49. Hardwares: Movidius Fathom
A USB stick for DNN
computations from Movidus
Movidius was purchased Intel recently

50. Hardware: Physical Neural Network
Started from 1960, PNN is
hardware electrically similar to
neural networks.
A BrainChip From NeuromorThings