The slides is about deep learning. (some contents is for japanese)

1. Before presentation…
I’ll presentation
Deep Learning and Extreme Learning Machine
In the first part, I’ll tell about theoretical topics
In the second part, I’ll tell about practical topics
It’s all about 40 minutes

2. Before presentation…
Deep Learning and Extreme learning machine is too tough…
I will tell about them without mathematic as much as possible
To be honest, I didn’t realize all of them, especially in mathematic
Instead of them, I’ll tell about practical topics
These are very more abstract
than many textbooks
(I’m sorry that the lack of knowledge)
For example, libraries, disciplines, and so on

3. Deep Learning
and
Extreme learning machine
(the first part)

4. Voice recognition Automatic translate Automatic driving
All these are enabled by “Deep Learning”
We use “future technologies”
“Deep Learning” attract many people

5. “Deep Learning” = deep neural network
What is “Deep Learning” ?
Perhaps you know already,
but I explain neural network easily
It’s said some type of things, but in this presentation…

6. Contents
Neural Network
Extreme Learning Machine
Different of two approaches

7. Neural Network
- chain of Neuron models -
Neural Network is a chain of neuron models
If the chain is deep,
the network is called “Deep Learning”
Neuron model
Neural Network

8. Neuron model imitates Neuron
Neural Network
- Neuron model -
Multiple input & Single output
(use very simple model)
Neuron model

9. Neural Network
- weight parameter -
Neuron model have weight parameter in every path
𝑤1
𝑤2
𝑤3
𝑥1 𝑤1 + 𝑥2 𝑤2 + 𝑥3 𝑤3Output:
When the time, the output is regarded as a multivariable function
The function can be analyzed by partial differentiation
𝜕𝑦
𝜕𝑤1
，
𝜕𝑦
𝜕𝑤2
，
𝜕𝑦
𝜕𝑤3
𝑦
𝑥1
𝑥2
𝑥3

10. Neural Network
- Can Neural Network be analyzed ? -
Can Neural Network be analyzed as neuron model ?
𝑤′1
𝑤′2
𝑤′3
𝑥1 𝑤1 + 𝑥2 𝑤2 + 𝑥3 𝑤3 =
𝜕𝑦
𝜕𝑤′2
𝑤1
𝑤2
𝑤3
𝑦
Yes, Neural Network can be analyzed recursively
𝑥1
𝑥2
𝑥3
As the example, neural network can be analyzed by partial differentiation

11. Neural Network
- Can Neural Network be analyzed ? -
 Neural Network is a partial differentiable function
(as I said)
 When the weight parameters are set correctly,
Neural Network can be every possible function
Neural Network is used
for unknown function approximation
Neural Network have two special qualities

12. Neural Network
- approximation -
loss function: 𝑓(𝑤)
𝑤
If you approximate Neural Network to unknown function,
you can use loss between output and ground truth as a function
loss
When the loss is minimum,
the weight parameter is the best
for approximation of the function
This point is purpose

13. Neural Network
- Feeling understanding -
𝜕𝑓(𝑤)
𝜕𝑤
loss
You can use the differentiation
for searching minimum point
loss function: 𝑓(𝑤)
𝑤
When you randomly set weight parameter,
you can know the differentiation of the loss function
𝜕𝑓(𝑤)
𝜕𝑤
Update weight !

14. Neural Network
- Feeling understanding -
𝜕𝑓(𝑤)
𝜕𝑤
loss
You can use the differentiation
for searching minimum point
loss function: 𝑓(𝑤)
𝑤
When you randomly set weight parameter,
you can know the differentiation of the loss function
𝜕𝑓(𝑤)
𝜕𝑤
Update weight !

15. Neural Network
- Feeling understanding -
𝜕𝑓(𝑤)
𝜕𝑤
loss
You can use the differentiation
for searching minimum point
loss function: 𝑓(𝑤)
𝑤
When you randomly set weight parameter,
you can know the differentiation of the loss function
𝜕𝑓(𝑤)
𝜕𝑤
Update weight !

16. Neural Network
- Feeling understanding -
𝜕𝑓(𝑤)
𝜕𝑤
loss
loss function: 𝑓(𝑤)
𝑤
Finally, you can get the best weight parameters
for approximation unknown function
For The best parameters,
update of parameters is
continued.
It’s too big order
Extreme
Learning Machine

17. Contents
Neural Network
Extreme Learning Machine
Different of two approaches

18. Extreme Learning machine
- Extreme Learning Machine’s idea -
𝜕𝑓(𝑤)
𝜕𝑤
loss
loss function: 𝑓(𝑤)
𝑤
Extreme Learning Machine’s idea is search minimum directly
For the best parameters,
update of parameters is
only one time.
It’s very small order
However, how can?

19. Extreme Learning Machine
- search minimum directly -
For example: simple Neural Network model
𝑦1
𝑥1
𝑥2
𝑥3
In Extreme Learning Machine,
vector 𝒚 transform vector 𝒕 by transformation matrix 𝜷
𝑦2
𝑡1
𝑡2
𝛽11
𝛽12
𝛽21
𝛽21

20. Extreme Learning Machine
- search minimum directly -
𝑦1
𝑥1
𝑥2
𝑥3
vector 𝒕 is indicated by vector 𝒚 and transformation matrix 𝜷
𝒕 = 𝒚𝜷
Therefore, transformation matrix 𝜷 is indicated
𝜷 = 𝒚+
𝒕
( 𝑦+: generalized inverse)
𝑦2
𝑡1
𝑡2
𝛽11
𝛽12
𝛽21
𝛽21

21. Extreme Learning Machine
- transformation matrix 𝜷 -
loss
loss function: 𝑓(𝑤)
𝑤
When the transformation matrix 𝜷 work in loss function,
depending on vector 𝒕, transformation matrix 𝜷 transform 𝒚
𝜷′′
𝜷′
𝜷′′′ 𝑦

22. Extreme Learning Machine
- transformation matrix 𝜷 -
loss
loss function: 𝑓(𝑤)
𝑤
Especially, when 𝒕 is ground truth,
transformation matrix 𝜷 transform 𝒚 to minimum
𝜷
𝜷′′
𝜷′′′
𝜷′
Therefore, Extreme
Learning Machine check
𝜷 only one time
when 𝒕 is ground truth

23. Contents
Neural Network
Extreme Learning Machine
Different of two approaches

24. Different of two approaches
- Extreme Leaning Machine’s advantage-
Extreme Learning Machine is better than Deep Leaning
Extreme Leaning Machine needn’t be differentiable
Extreme Leaning Machine doesn’t use differentiation,
so the function needn’t be differentiable function as Deep Leaning
Extreme Leaning Machine need few parameters
Deep Leaning need many parameters for update parameters,
but Extreme Leaning Machine update only one time
Extreme Leaning Machine learn firstly

25. Different of two approaches
- Extreme Learning Machine’s weak points -
Extreme Learning Machine sounds better than Deep Learning,
but Extreme Learning Machine has some weak points
 Extreme Learning Machine fall into a local solution
Deep Leaning can escape from local solutions in enough updates,
but Extreme Leaning Machine depends on initialization
 Extreme Learning Machine can’t be a cat‐and‐mouse game
Some of Deep Leanings need a cat-and-mouse game
in learning process, but Extreme Leaning Machine can’t be enough

26. Different of two approaches
- Extreme Learning Machine’s weak points -
However, I think it’s more important than these
that Deep Learning can be coded with libraries easily
Extreme Learning Machine sounds better than Deep Learning,
but Extreme Learning Machine has some weak points
 Extreme Learning Machine fall into a local solution
Deep Leaning can escape from local solutions in enough updates,
but Extreme Leaning Machine depends on initialization
 Extreme Learning Machine can’t be a cat‐and‐mouse game
Some of Deep Leanings need a cat-and-mouse game
in learning process, but Extreme Leaning Machine can’t be enough

27. Different of two approaches
- Extreme Learning Machine’s weak points -
However, I think it’s more important than these
that Deep Learning can be coded with some libraries
Extreme Learning Machine sounds better than Deep Learning,
but Extreme Learning Machine has some weak points
 Extreme Learning Machine fall into a local solution
Deep Leaning can escape from local solutions in enough updates,
but Extreme Leaning Machine depends on initialization
 Extreme Learning Machine can’t be a cat‐and‐mouse game
Some of Deep Leanings need a cat-and-mouse game
in learning process, but Extreme Leaning Machine can’t beIt’s the second half

28. If you have question,
please tell me about them
END
(The first part)

29. More topics for questions
- Actually, is this Deep Learning ? -
For explaining easily, I left out some topics
Ex: collect artificial Neuron model
𝑦
𝑥1
𝑥2
𝑥3
bias
(It’s special weight for steady learning)
activation function
(it make Neural Network to almighty)
1

30. More topics for questions
- Why isn’t inverse used in Deep Learning -
Because the order is too high for big data
Every layer’s partial differentiation is in 𝑂(𝑛),
but generalized inverse is in more than 𝑂(𝑛3
)
Due to same reason,
second order differential isn’t used in Deep Learning
**As this pages source isn’t enough, I may be wrong it**

32. Deep Learning
and
Extreme learning machine
(the second part)
Yoshihiro Yamada
(Student ID:2150104071)

33. Do you remember it ?
1st, I’ll tell about this!
I said in finial slide of the first half…

34. Deep Learning libraries
- Chainer’s example -
For example, in Chainer that is a Deep Leaning library,
Deep Leaning can be coded as below code
Nowadays, Deep Learning coding is
easier than Extreme Leaning Machine
Initialize model
define flow of
computing model
define loop of training

35. Contents
Deep Learning libraries
Graphical Processing Unit
If you want start now
If you got environment…

36. Deep Learning libraries
- The origin of Deep Leaning libraries -
In Deep Leaning,
differentiations have to be defined
It’s very high costs for coding,
but mainly differentiations is common
Some libraries are made for Deep Leaning
Theano, Torch, Caffe, Chainer, TensorFlow and so on…

37. Deep Learning libraries
- Many libraries -
I’ll explain some libraries in these
1st generation:
2nd generation:
3rd generation:
Too many libraries for Deep Learning now

38. Deep Learning libraries
- Deferent of main libraries -
Torch (language: Lua)
Very fast library
Have multifunction for Machine Leaning
Standard Library for Deep Leaning

39. Deep Learning libraries
- Deferent of main libraries -
Chainer (language: python)
Very easy for understanding
Bug fix is easiest in this library
The most popular library in Japan

40. Deep Learning libraries
- Deferent of main libraries -
TensorFlow (language: python)
Made by Google
Easy for understanding
Booming library in the world
• But…, I think it’s worse than Chainer in all sides

41. Deep Learning libraries
- Deferent of main libraries -
Nervana (language: python)
Fastest libraries in single machine
• In a benchmarks, this library is ¾ winner
But, I didn’t touch the library
• I didn’t now much things about this one

42. Deep Learning libraries
- Deferent of main libraries -
In conclusion, If you want to …
 check many examples: Torch
 make Deep Leaning work, anyway: Chainer
 use library with many materials: TensorFlow
 use fastest library: Nervana

43. Deep Learning libraries
- Common things of main libraries -
All these libraries have two functions
1. (of course) Deep Leaning
2. General-Purpose computing on Graphics Processing Units
Deep Leaning need it for fast computing, now
Why does GPU use?

44. Contents
Deep Learning libraries
Graphical Processing Unit
If you want start now
If you got environment…

45. Graphical Processing Unit
- architecture -
Graphical Processing Unit (GPU)
 A Processing Unit for drawing graphics
 Compute parallel and speedy for graphics
 Every core’s cost is cheaper than CPU
If we set correctly,
GPU can be handled as cheap parallel computer
It’s reason why GPU is standard in Deep Leaning

46. Graphical Processing Unit
- General-Purpose computing on GPU -
single CPU vs single GPU
Actually, in my test of computing speed
× 16 faster
winner

47. Graphical Processing Unit
- General-Purpose computing on GPU -
Nowadays, GPU is cursed…
(“TITAN X”, the highest GPU)
Because it have only 12GB memory
Therefore, almost Deep Leaning limit is in 12GB
(It's just an aside)

48. Graphical Processing Unit
- General-Purpose computing on GPU -
I’ll sell new highest GPU
in this year !GPU maker
Perhaps, this curse is broken in this year…
The year to make things happen

49. Contents
Deep Learning libraries
Graphical Processing Unit
If you want start Deep Leaning now
• I’ll tell about money…
If you got environment…

50. If you want to start now
OS (Ubuntu) 0
GPU (TITAN X × 1) 150,000
Base PC’s price 150,000
total 300,000
If you want the highest environment
× 30You can get the environment by

51. If you want to start now
× 30
I am normal poor student, so it’s too expensive to buy
(Therefore, I use server in my laboratory now)
However, if you have some money for some reasons…
I think it’s not so expensive
I think I can’t do, but I don’t think you can’t do,
because you check these kind of slides
Advertisement

52. Contents
Deep Learning libraries
Graphical Processing Unit
If you want start now
If you got environment…
• In this final section, I’ll explain about researches
in nowadays
• All members will be already exhausted, so I’ll
use figures or movies as much as possible

53. If you got environment…
Residual Network (ResNet)
 The strongest CI for image classification
 Deep, so deep…
AlexNet (2012)
8 layers ResNet (2015)
152 layers

54. If you got environment…
Deep Q-Networks (DQN)
 Reinforcement learning with Deep Leaning
 Can play simple games better than human

55. If you got environment…
Deep Q-Networks (DQN)
 Reinforcement learning with Deep Leaning
 Can play simple games better than human
https://www.youtube.com/watch?v=a3AWpeOjkzw
Deep Q-Networks auto driving demonstration (Japanese)
DQN is been
putting into use

56. If you got environment…
Deep Convolutional Generative Adversarial Networks
(DCGAN)
 Learn in a cat‐and‐mouse game for drawing images
 Can compute image operation by class words, finally
Faker CI
VS
Appraiser CI
Fake

57. If you got environment…
Spatial Transformer Networks (STN)
 Learn cutting and classification images
in one network
Hand written characters recognition in STN

58. Conclusion

59. Conclusion
Today, I said many things
Theoretical things of NN and ELM
Environment of libraries and GPU
Interesting research topics in nowadays
However, I couldn’t tell all of these things
In the end, I introduce some things for learning more

60. Conclusion
If you want more theoretical things
『深層学習』
Main Deep Leaning technics and knowledge
are written in the book smartly
『パターン認識と機械学習』
Almost mathematic background in the books
(but I gave up because too high level)
(Japanese)
(English/Japanese)

61. Conclusion
If you want more practical things
『サクっとディープラーニングくん』 (my product)
 I set some main function
for image classification
 I hope it’s used for
Deep Learning beginner
(English/Japanese)

62. If you have question,
please tell me about them
END

63. More topics for questions
- What layers these researches have ? -
Deep Q-Networks (DQN)
4 layers (convolution 2 - full chained 2)
Convolution ?
A neuron model which don’t have all path
It’s used for dealing with
rich data as images

64. More topics for questions
- What layers these researches have ? -
Deep Q-Networks (DQN)
4 layers (convolution 2 - full chained 2)
Spatial Transformer Networks (STN)
Free (in theise, used 4 layers [cov 2 - fc 2] in ST)
Deep Convolutional GAN (DCGAN)
Faker: 5 layers , Appraiser: 5 layers

66. Reference
• Constrained Extreme Learning Machines: A Study on Classification Cases
(http://arxiv.org/ftp/arxiv/papers/1501/1501.06115.pdf)
• Deep Residual Learning for Image Recognition
(http://arxiv.org/pdf/1512.03385v1.pdf)
• Playing Atari with Deep Reinforcement Learning
(http://arxiv.org/pdf/1312.5602.pdf)
• Unsupervised Representation Learning with Deep Convolutional Generative
Adversarial Networks
(http://arxiv.org/pdf/1511.06434v2.pdf)
• Spatial Transformer Networks
(http://arxiv.org/pdf/1506.02025v2.pdf)

67. Reference
• Extreme Learning Machine – Tech.D-ITlab | Denso IT
Laboratory researcher's blog sites
(https://tech.d-itlab.co.jp/ml/229/)
• Convnet-benchmarks
(https://github.com/soumith/convnet-
benchmarks/blob/master/README.md#one-small-note)
• Deep learning Libs @twm
(http://www.slideshare.net/yutakashino/deep-learning-libs-twm)

68. Reference
• 分散深層強化学習でロボット制御
(https://research.preferred.jp/2015/06/distributed-deep-reinforcement-
learning/)
• DQNの生い立ち ＋ Deep Q-NetworkをChainerで書いた
(http://qiita.com/Ugo-Nama/items/08c6a5f6a571335972d5)
• Chainerを使ってコンピュータにイラストを描かせる
(http://qiita.com/rezoolab/items/5cc96b6d31153e0c86bc)
• 深層学習と機械知覚
(http://randomfield.cs.waseda.ac.jp/files/nakayama.pdf)