Urs Köster - Convolutional and Recurrent Neural Networks

1. Deep Learning Deep Dive & Workshop on Convolutional and Recurrent Neural Networks Urs Köster San Diego Deep Learning Meetup December 2, 2015

2. Outline 2 • Deep Learning • Nervana • Neon • Convolutional Network Demo (meetup.nervanasys.com) • Recurrent Network Demo (meetup.nervanasys.com)

3. INTRO TO DEEP LEARNING 3

4. 4 Scene Parsing *Yann LeCun https://www.youtube.com/watch?v=ZJMtDRbqH40

5. 5 Speech Translation *Skype https://www.youtube.com/watch?v=eu9kMIeS0wQ

6. 6 Understanding Images *Karpathy http://cs.stanford.edu/people/karpathy/deepimagesent/

7. 7 What is deep learning? Historical perspective: • Input → designed features → output • Input → designed features → SVM → output • Input → learned features → SVM → output • Input → levels of learned features → output

8. 8 What is deep learning? A method for extracting features at multiple levels of abstraction • Features are discovered from data • Performance improves with more data • Network can express complex transformations • High degree of representational power

9. 9 What is deep learning? No free lunch: • lots of data • flexible models • powerful priors

10. 10 Imagenet ILSVRC Challenge Error rate1 Source: ImageNet 1: ImageNet top 5 error rate 0% 10% 20% 30% 2010 2011 2012 2013 2014 2015 Deep learning techniques human performance

11. NERVANA PLATFORM 11

12. 12 nervana platform for deep learning neon deep learning framework train deploy nervana cloud explore GPUs CPUs nervana engine AWS VM S3 S3 Web VM VM VM VM VM S3

13. 13 Deep learning as a core technology DL Photos Maps Voice Search Self-driving car Ad Targeting Machine Translation ‘Google Brain’ model DL Image classification Image localization Speech recognition Video indexing Sentiment analysis Machine Translation Nervana Platform

14. • Architecture optimized for algorithm 14 Core technology • Unprecedented compute density • Scalable distributed architecture • Learning and inference

15. 15 Markets Pharma Oil&Gas AgricultureMedical $ Finance Internet Govt

16. NEON 16

17. neon: nervana python deep learning library 17 • User-friendly, extensible, abstracts parallelism & data caching • Support for many deep learning models • Interface to nervana cloud • Multiple backends • nervana engine • GPU (optimized on assembler level) • CPU cluster Open source (Apache 2.0) on github.com/nervanaSystems/neon

18. 18 Image classification (VGG-D) speed comparison Speed for one full forward/backward pass on VGG model D Imagespersecond 0 20 40 60 80 100 120 Neon Caffe • Neon trains networks about 2x faster! • Caffe uses CuDNN v3 (NVidia’s own optimized library) • Same holds for other models (GoogLeNet, AlexNet) • And other Frameworks (Torch7, TensorFlow)

19. 1 Soumith Chintala, github.com/soumith/convnet-benchmarks Benchmarks for convnets1 19 Benchmarks compiled by Facebook. Smaller is better.

20. 20 End-to-end optimized • GPU Kernels: Written in SASS Assembler, near full utilization for most layers • Data Loader: neon never blocks waiting for data Library Wrapper DataLoader DataLoader DecodeThreads start IOThreads destroy thread pool stop next ... next create thread pool create thread pool destroy thread pool read macrobatch file decode decode decode macrobatch buffers minibatch buffers (pinned) raw file buffers Control Codes Dual issue instr. Fused fp32 multiply add Load from shared Barrier sync Set barrier

21. Proprietary and conﬁdential. Do not distribute. 21 Running locally: % python rnn.py # or neon rnn.yaml Running in nervana cloud: % ncloud submit rnn.py # or rnn.yaml % ncloud show <model_id> % ncloud list % ncloud deploy <model_id> % ncloud predict <model_id> <data> # or use REST api

22. 22 HANDS ON! • Option 1: Interactive ipython notebook in your browser without installing anything (you should have paper slip with your URL and password) • Option 2: Use your linux or mac laptop to download neon and try our examples. Get the notebooks and data from meetup.nervanasys.com

23. Contact 23 urs@nervanasys.com github.com/NervanaSystems/neon

Urs Köster - Convolutional and Recurrent Neural Networks

Nervana Systems

Urs Köster - Convolutional and Recurrent Neural Networks