Download as PDF, PPTX
![Logistic sigmoid function
• Very widespread
• Delimits output to [0, 1]
• Vanishing gradient
problem](https://image.slidesharecdn.com/deeplearning-180427063627/85/Deep-learning-a-primer-48-320.jpg)
![Hyperbolic tangent
• Very widespread
• Delimits output to [-1, 1]
• Vanishing gradient
problem](https://image.slidesharecdn.com/deeplearning-180427063627/85/Deep-learning-a-primer-49-320.jpg)
Deep learning - a primer
- 1. Deep learning A primerfor the curious developer Uwe Friedrichsen & Dr. Shirin Glander –codecentric AG – 2018
- 2. @ufried Uwe Friedrichsen uwe.friedrichsen@codecentric.de @ShirinGlander Dr. ShirinGlander shirin.glander@codecentric.de
- 3. Why should Icare about Deep Learning?
- 4. Deep learning hasthe potential to affect white collar workers (including IT) in a similar way as robots affected blue collar workers.
- 5. What is DeepLearning? Some success stories
- 16. What is DeepLearning? A rough classification
- 17.
- 18. Traditional AI Focus onproblems that are ... • ... hard for humans • ... straightforward for computers • ... can be formally described Deep Learning Focus on problems that are ... • ... intuitive for humans • ... difficult for computers (hard to be described formally) • ... best learnt from experience
- 19. Where does DeepLearning come from?
- 20. General evolution • Twoopposed forces • Recreation of biological neural processing • Abstract mathematical models (mostly linear algebra) • Results in different models and algorithms • No clear winner yet
- 21. Cybernetics (ca. 1940- 1960) • ADALINE, Perceptron • Linear models, typically no hidden layers • Stochastic Gradient Descent (SGD) • Limited applicability • E.g., ADALINE could not learn XOR • Resulted in “First winter of ANN” (Artificial Neural Networks)
- 22. Connectionism (ca. 1980- 1990) • Neocognitron • Non-linear models, distributed feature representation • Backpropagation • Typically 1, rarely more hidden layers • First approaches of sequence modeling • LSTM (Long short-term memory) in 1997 • Unrealistic expectations nurtured by ventures • Resulted in “Second winter of ANN”
- 23. Deep Learning (ca.2006 -) • Improved algorithms, advanced computing power • Enabled training much larger and deeper networks • Enabled training much larger data sets • Typically several to many hidden layers • Overcame the “feature extraction dilemma”
- 24. What is DeepLearning used for?
- 25. Deep Learning applicationareas • Classification (incl. missing inputs) • Regression (value prediction) • Function prediction • Density estimation • Structured output (e.g., translation) • Anomaly detection • Synthesis and sampling • Denoising • Compression (dimension reduction) • ...
- 26. How does DeepLearning work? A first (scientifically inspired) approach
- 27. „A computer programis said to learn • from experience E • with respect to some class of tasks T • and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” -- T. Mitchell, Machine Learning, p. 2, McGraw Hill (1997) Supervised learning, unsupervised learning, reinforcement learning, ... Too difficult to solve with fixed programs designed by humans Accuracy vs. error rate, training vs. test set, ...
- 28. Err ... Hmm ... Well... I don’t get it!
- 29. How does DeepLearning work? A second (more down-to-earth) approach
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. Neuron • Design inspiredby biological neurons • One or more inputs • Processing (and state storage) unit • One or more outputs • In practice often implemented as tensor transformations • Relevance of internal state depends on network type • Usually negligible for feed-forward networks • Usually relevant for recurrent networks Neuron Processing (+ State) Output(s) Input(s) ... ...
- 37.
- 38. Layer • Neurons typicallyorganized in layers • Input and output layer as default • Optionally one or more hidden layer • Layer layout can have 1-n dimensions • Neurons in different layers can have different properties • Different layers responsible for different (sub-)tasks Output layer Input layer ... N 1 2 Hidden layer(s) ...
- 39.
- 40. Connection • Usually connectinput and output tensor in a 1:1 manner • Connect between layers (output layer N-1 à input layer N) • Layers can be fully or partially (sparsely) connected • RNNs also have backward and/or self connections • Some networks have connections between neurons of the same layer (e.g., Hopfield nets, Boltzmann machines) Input tensor(s) Output tensor(s)
- 41.
- 42. Weight • (Logically) augmentsa connection • Used to amplify or dampen a signal sent over a connection • The actual “memory” of the network • The “right” values of the weights are learned during training • Can also be used to introduce a bias for a neuron • By connecting it to an extra neuron that constantly emits 1 W Weight
- 43.
- 44. Input tensor(s) Output tensor(s) Step1 • For each neuron of input layer • Copy resp. input tensor’s value to neuron’s input • Calculate state/output using activation function (typically linear function, passing value through) Step 2-N • For each hidden layer and output layer in their order • For each neuron of the layer • Calculate weighted sum on inputs • Calculate state/output using activation function (see examples later) Final step • For each neuron of output layer • Copy neuron’s output to resp. output tensor’s value
- 45. Input tensor(s) Output tensor(s) Step1 Final step Step 2-N • Default update procedure (most widespread) • All neuron per layer in parallel • Different update procedures exist • E.g., some Hopfield net implementations randomly pick neurons for update
- 46.
- 47. Linear function • Easyto handle • Cannot handle non-linear problems
- 48. Logistic sigmoid function • Very widespread • Delimits output to [0, 1] • Vanishing gradient problem
- 49. Hyperbolic tangent • Verywidespread • Delimits output to [-1, 1] • Vanishing gradient problem
- 50. Rectified linear unit(ReLU) • Easy to handle • No derivative in 0 • Dying ReLU problem • Can be mitigated, e.g., by using leaky ReLU
- 51. Softplus • Smooth approximation ofReLU • ReLU usually performs better • Thus, use of softplus usually discouraged
- 52.
- 53. Hyperparameter • Influence networkand algorithm behavior • Often influence model capacity • Not learned, but usually manually optimized • Currently quite some research interest in automatic hyperparameter optimization Examples • Number of hidden layers • Number of hidden units • Learning rate • Number of clusters • Weight decay coefficient • Convolution kernel width • ...
- 54.
- 55.
- 56.
- 57. Cost function (a.k.a.loss function) • Determines distance from optimal performance • Mean squared error as simple (and widespread) example
- 58. Cost function (a.k.a.loss function) • Determines distance from optimal performance • Mean squared error as simple (and widespread) example • Often augmented with regularization term for better generalization (see challenges)
- 59.
- 60.
- 61. Stochastic gradient descent • Direct calculation of minimum often not feasible • Instead stepwise “descent” using the gradient à Gradient descent
- 62. Stochastic gradient descent • Direct calculation of minimum often not feasible • Instead stepwise “descent” using the gradient à Gradient descent • Not feasible for large training sets • Use (small) random sample of training set per iteration à Stochastic gradient descent (SGD)
- 63.
- 64. Stochastic gradient descent x ε* gradient x’ Learning rate ε
- 65.
- 66. Backpropagation • Procedure tocalculate new weights based on loss function Depends on cost function Depends on activation function Depends on input calculation
- 67. Backpropagation • Procedure tocalculate new weights based on loss function • Usually “back-propagated” layer-wise • Most widespread optimization procedure Depends on cost function Depends on activation function Depends on input calculation
- 68.
- 69. Training set Validation/ Test set Data Training Quality measure Stochastic gradient descent Back- propagation Cost function Optimization procedure
- 70. Data set • Consistsof examples (a.k.a. data points) • Example always contains input tensor • Sometimes also contains expected output tensor (depending on training type) • Data set usually split up in several parts • Training set – optimize accuracy (always used) • Test set – test generalization (often used) • Validation set – tune hyperparameters (sometimes used)
- 71. Data Types Training Quality measure Stochastic gradient descent Back- propagation Training set Validation/ Testset Cost function Optimization procedure
- 72. Supervised Data Types Training Quality measure Stochastic gradient descent Back- propagation Training set Validation/ Testset Cost function Optimization procedure
- 73. Supervised learning • Typicallylearns from a large, yet finite set of examples • Examples consist of input and output tensor • Output tensor describes desired output • Output tensor also called label or target • Typical application areas • Classification • Regression and function prediction • Structured output problems
- 74. Unsupervised Data Types Supervised Training Quality measure Stochastic gradient descent Back- propagation Training set Validation/ Testset Cost function Optimization procedure
- 75. Unsupervised learning • Typicallylearns from a large, yet finite set of examples • Examples consist of input tensor only • Learning algorithm tries to learn useful properties of the data • Requires different type of cost functions • Typical application areas • Clustering, density estimations • Denoising, compression (dimension reduction) • Synthesis and sampling
- 76. Reinforcement Data Types Supervised Training Quality measure Unsupervised Stochastic gradient descent Back- propagation Training set Validation/ Testset Cost function Optimization procedure
- 77. Reinforcement learning • Continuouslyoptimizes interaction with an environment based on reward-based learning Agent Environment State t Reward t State t+1 Reward t+1 Action t
- 78. Reinforcement learning • Continuouslyoptimizes interaction with an environment based on reward-based learning • Goal is selection of action with highest expected reward • Takes (discounted) expected future rewards into account • Labeling of examples replaced by reward function • Can continuously learn à data set can be infinite • Typically used to solve complex tasks in (increasingly) complex environments with (very) limited feedback
- 79. Challenges Data Types Supervised Training Quality measure Unsupervised Reinforcement Stochastic gradient descent Back- propagation Training set Validation/ Testset Cost function Optimization procedure
- 80. Data Types Supervised Training Quality measure Unsupervised Reinforcement Stochastic gradient descent Back- propagation Under-/ Overfitting Training set Validation/ Testset Cost function Challenges Optimization procedure
- 81. Underfitting and Overfitting • Training error describes how good training data is learnt • Test error is an indicator for generalization capability • Core challenge for all machine learning type algorithms 1. Make training error small 2. Make gap between training and test error small • Underfitting is the violation of #1 • Overfitting is the violation of #2
- 82. Good fit Underfitting Overfitting Trainingdata Test data
- 83. Underfitting and Overfitting • Under- and overfitting influenced by model capacity • Too low capacity usually leads to underfitting • Too high capacity usually leads to overfitting • Finding the right capacity is a challenge
- 84. Data Types Supervised Training Quality measure Unsupervised Reinforcement Stochastic gradient descent Back- propagation Under-/ Overfitting Training set Validation/ Testset Cost function Regularization Challenges Optimization procedure
- 85. Regularization • Regularization isa modification applied to learning algorithm • to reduce the generalization error • but not the training error • Weight decay is a typical regularization measure
- 86. Data Types Supervised Training Quality measure Unsupervised Reinforcement Stochastic gradient descent Back- propagation Under-/ Overfitting Transfer learning Training set Validation/ Testset Cost function Regularization Challenges Optimization procedure
- 87. Transfer learning • Howto transfer insights between related tasks • E.g., is it possible to transfer knowledge gained while training to recognize cars on the problem of recognizing trucks? • General machine learning problem • Subject of many research activities
- 88.
- 89.
- 90. Multilayer perceptron (MLP) • Multilayer feed-forward networks • “Vanilla” neural networks • Typically used for • Function approximation • Regression • Classification Image source: https://deeplearning4j.org
- 91.
- 92. Convolutional neural network(CNN) • Special type of MLP for image processing • Connects convolutional neuron only with receptive field • Advantages • Less computing power required • Often even better recognition rates • Inspired by organization of visual cortex Image source: https://deeplearning4j.org
- 93.
- 94. Recurrent neural network(RNN) • Implements internal feedback loops • Provides a temporal memory • Typically used for • Speech recognition • Text recognition • Time series processing Image source: https://deeplearning4j.org
- 95.
- 96. Long short-term memory(LSTM) • Special type of RNN • Uses special LSTM units • Can implement very long-term memory while avoiding the vanishing/exploding gradient problem • Same application areas as RNN Image source: https://deeplearning4j.org
- 97.
- 98. Autoencoder • Special typeof MLP • Reproduces input at output layer • Consists of encoder and decoder • Usually configured undercomplete • Learns efficient feature codings • Dimension reduction (incl. compression) • Denoising • Usually needs pre-training for not only reconstructing average of training set Image source: https://deeplearning4j.org
- 99.
- 100. Generative adversarial networks(GAN) • Consists of two (adversarial) networks • Generator creating fake images • Discriminator trying to identify fake images • Typically used for • Synthesis and sampling (e.g., textures in games) • Structured output with variance (e.g., variations of a design or voice generation) • Probably best known for creating fake celebrity images Image source: https://deeplearning4j.org
- 101.
- 102. How does DeepLearning feel in practice?
- 132. What issues mightI face if diving deeper?
- 133. Issues you mightface • Very fast moving research domain • You need the math. Really! • How much data do you have? • GDPR: Can you explain the decision of your network? • Meta-Learning as the next step • Monopolization of research and knowledge
- 134.
- 135. Wrap-up • Broad, diversetopic • Very good library support and more • Very active research topic • No free lunch • You need the math! à Exciting and important topic – become a part of it!
- 136. References • I. Goodfellow,Y. Bengio, A. Courville, ”Deep learning", MIT press, 2016, also available via https://www.deeplearningbook.org • C. Perez, “The Deep Learning AI Playbook”, Intuition Machine Inc., 2017 • F. Chollet, "Deep Learning with Python", Manning Publications, 2017 • OpenAI, https://openai.com • Keras, https://keras.io • Deep Learning for Java, https://deeplearning4j.org/index.html • Deep Learning (Resource site), http://deeplearning.net
- 137. @ShirinGlander Dr. Shirin Glander shirin.glander@codecentric.de @ufried UweFriedrichsen uwe.friedrichsen@codecentric.de