The document discusses the evolution and application of deep learning and neural networks, highlighting its ability to process large amounts of data and learn complex problems. It details the historical development from early perceptrons to advanced algorithms and architectures that can handle significant computational tasks, facilitated by the rise of GPUs. It also emphasizes the practical applications of deep learning in various fields while acknowledging the challenges and intricacies involved in its implementation.

1. Neural Networks
and Deep Learning
Tijmen Blankevoort

2. Scyfer
Prof dr. Max Welling
Drs. Jorgen Sandig
Msc. Taco Cohen

3. Deep Learning
All purpose machine learning
Using Neural Networks:
- Using large amounts of data
- Learning very complex problems
- Automatically learning features
A new era of machine learning

4. Deep learning wins all competitions
- IJCNN 2011 Traffic Sign Recognition Competition
- ISBI 2012 Segmentation of neuronal structures in EM
stacks challenge
- ICDAR 2011 Chinese
handwriting recognition

5. Applications
A lot of state of the art systems use deep learning to some
extent:
- IBMs Watson: Jeopardy contest 2011
- Google’s self-driving car
- Google Glasses
- Facebook face recognition
- Facebook user modelling
Mostly image and sound recognition tasks (difficult)

6. Google Brain (2011)
- 10 million youtube/imagenet images
- 1 billion parameters
- 16.000 processors
- Largely unsupervised!
- 20.000 categories
- 15.8% accuracy

7. Bigger, better
Deep Learning:
- The scope of what
computers can learn has
greatly been increased
- Interaction with the real
world

8. Biological Inspiration
Neuron

9. Neuron computer model

10. Activation function
Sigmoid activation function

11. Neuron computer model
Perceptron - 1957 Rosenblatt

12. Easy functions with a neuron

13. Linking neurons and training
- Initialize randomly
- Sequentially give it data.
- See what the difference is between
network output and actual output.
- Update the weights according to this error.
- End result: give a model input, and it
produces a proper output.
Quest for the weights. The weights are the
model!

14. The Perceptron (1958)
“A machine which senses, recognizes, remembers, and responds like the human mind”
“Remarkable machine… [was] capable of what amounts to thought” - The New Yorker

15. Criticism and downfall (1969)
- Perceptrons are painfully limited. They can not even learn a
simple XOR function!
- No feasible way of learning networks with multiple layers
- Interest in neural networks close to fully disappeared

16. Renewed interest (90’s)
- Learning multiple layers
- “Back propagation”
- Can theoretically learn any
function!
But…
Very slow and inefficient
- Machine learning attention
towards SVMs, random forests
etc.

17. Deep learing (2006)
- Quest: Mimic human brain representations
- Large networks
- Lots of data
Problem:
Simple back propagation fails
on large networks.

18. Deep learning (2006)
- Exactly same networks as
before, just BIGGER
- Combination of three factors:
- (Big data)
- Better algorithms
- Parallel computing (GPU)

19. Better algorithms
Restricted Boltzmann machine
Pre-training: Learn the representation by parts!
Very strong unsupervised learning
After pre-training, use back propagation

20. Parallel (GPU) power
- Every set of weights can be stored as a matrix (w_ij)
- GPUs are made to do common parallel problems fast!
- All similar calculations done at the same time, huge performance boost.
- CPU parallelizing

21. Future of Deep Learning
- Currently an explosion of developments
- Hessian-Free networks (2010)
- Long Short Term Memory (2011)
- Large Convolutional nets, max-pooling (2011)
- Nesterov’s Gradient Descent (2013)
- Currently state of the art but...
- No way of doing logical inference (extrapolation)
- No easy integration of abstract knowledge
- Hypothetic space bias might not conform with reality

22. When to apply Deep Learning
- Generally, vision and sound
recognition, but...
- Works great for any other problem too!
- A lot of data / features
- Don’t want to make your own features
- State of the art results

23. How to apply Deep Learning
Deep learning is very difficult!
- No easy plug and play software
- Far too many different networks/options/additions
- Mathematics and programming very challenging
- Research is fast paced
- Learning a network is both an art and a science
My advice:
Cooperation university <=> business

24. How to apply Deep Learning
- For most current business problems, no need for
expensive hardware. e.g. we use a laptop