6. GPU Revolution
For 30 years, the dynamics of Moore’s Law held true as microprocessor
performance grew at 50 percent per year. But the limits of semiconductor
physics mean that CPU performance now only grows by 10 percent per year.
NVIDIA GPU computing has given the industry a path forward — and will
provide a 1,000X speed-up by 2025.
7. Why GPU ?
• GPU can perform rasterization very fast for gaming visualization.
• The same power is utilized for independent calculations in DNN.
• But there are some trade-offs of using GPU.
13. Recurrent Neural Network (CNN)
• Vanishing/Exploding Gradient
• Yoshua Bengio (2010)
– We need the variance of the outputs of each layer to be equal to the
variance of its inputs
14. Recurrent Neural Network (CNN)
• Batch Normalization (2015)
– The vanishing gradient can come back
– adding an operation in the model just before the activation function of
each layer, simply zero-centering and normalizing the inputs, then scaling
and shifting the result using two new parameters per layer
15. Convolutional Neural Network (CNN)
• CNN are used for higher order tensors (images in computer vision)
• Motivation
– Reduces the number of variables
– Can detect spatial features
– Fixed input size
• Edge Detection
– Sobel Filter
– Prewitt Filter
16. Convolutional Neural Network (CNN)
• Convolution Example
• Types
– Valid : No Padding
– Same : Padding
• Stride
• Activation Function
– Tanh is used in most of the applications
20. Convolutional Neural Network (CNN)
• LeNet-5
– The LeNet-5 architecture is perhaps the most widely known CNN
architecture.
– It was created by Yann LeCun in 1998 and widely used for hand‐written
digit recognition (MNIST)
