Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neural networks and deep learning


Published on

Tijmen Blankenvoort, co-founder Scyfer BV, presentation at Artificial Intelligence Meetup 15-1-2014. Introduction into Neural Networks and Deep Learning.

Published in: Technology, Education

Neural networks and deep learning

  1. 1. Neural Networks and Deep Learning Tijmen Blankevoort
  2. 2. Scyfer Prof dr. Max Welling Drs. Jorgen Sandig Msc. Taco Cohen
  3. 3. Deep Learning All purpose machine learning Using Neural Networks: - Using large amounts of data - Learning very complex problems - Automatically learning features A new era of machine learning
  4. 4. Deep learning wins all competitions - IJCNN 2011 Traffic Sign Recognition Competition - ISBI 2012 Segmentation of neuronal structures in EM stacks challenge - ICDAR 2011 Chinese handwriting recognition
  5. 5. Applications A lot of state of the art systems use deep learning to some extent: - IBMs Watson: Jeopardy contest 2011 - Google’s self-driving car - Google Glasses - Facebook face recognition - Facebook user modelling Mostly image and sound recognition tasks (difficult)
  6. 6. Google Brain (2011) - 10 million youtube/imagenet images - 1 billion parameters - 16.000 processors - Largely unsupervised! - 20.000 categories - 15.8% accuracy
  7. 7. Bigger, better Deep Learning: - The scope of what computers can learn has greatly been increased - Interaction with the real world
  8. 8. Biological Inspiration Neuron
  9. 9. Neuron computer model
  10. 10. Activation function Sigmoid activation function
  11. 11. Neuron computer model Perceptron - 1957 Rosenblatt
  12. 12. Easy functions with a neuron
  13. 13. Linking neurons and training - Initialize randomly - Sequentially give it data. - See what the difference is between network output and actual output. - Update the weights according to this error. - End result: give a model input, and it produces a proper output. Quest for the weights. The weights are the model!
  14. 14. The Perceptron (1958) “A machine which senses, recognizes, remembers, and responds like the human mind” “Remarkable machine… [was] capable of what amounts to thought” - The New Yorker
  15. 15. Criticism and downfall (1969) - Perceptrons are painfully limited. They can not even learn a simple XOR function! - No feasible way of learning networks with multiple layers - Interest in neural networks close to fully disappeared
  16. 16. Renewed interest (90’s) - Learning multiple layers - “Back propagation” - Can theoretically learn any function! But… Very slow and inefficient - Machine learning attention towards SVMs, random forests etc.
  17. 17. Deep learing (2006) - Quest: Mimic human brain representations - Large networks - Lots of data Problem: Simple back propagation fails on large networks.
  18. 18. Deep learning (2006) - Exactly same networks as before, just BIGGER - Combination of three factors: - (Big data) - Better algorithms - Parallel computing (GPU)
  19. 19. Better algorithms Restricted Boltzmann machine Pre-training: Learn the representation by parts! Very strong unsupervised learning After pre-training, use back propagation
  20. 20. Parallel (GPU) power - Every set of weights can be stored as a matrix (w_ij) - GPUs are made to do common parallel problems fast! - All similar calculations done at the same time, huge performance boost. - CPU parallelizing
  21. 21. Future of Deep Learning - Currently an explosion of developments - Hessian-Free networks (2010) - Long Short Term Memory (2011) - Large Convolutional nets, max-pooling (2011) - Nesterov’s Gradient Descent (2013) - Currently state of the art but... - No way of doing logical inference (extrapolation) - No easy integration of abstract knowledge - Hypothetic space bias might not conform with reality
  22. 22. When to apply Deep Learning - Generally, vision and sound recognition, but... - Works great for any other problem too! - A lot of data / features - Don’t want to make your own features - State of the art results
  23. 23. How to apply Deep Learning Deep learning is very difficult! - No easy plug and play software - Far too many different networks/options/additions - Mathematics and programming very challenging - Research is fast paced - Learning a network is both an art and a science My advice: Cooperation university <=> business
  24. 24. How to apply Deep Learning - For most current business problems, no need for expensive hardware. e.g. we use a laptop