Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[2A4]DeepLearningAtNAVER

18,411 views

Published on

DEVIEW 2014 [2A4]DeepLearningAtNAVER

Published in: Technology
  • -- DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT -- ......................................................................................................................... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... (Unlimited)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If you want to download or read this book, Copy link or url below in the New tab ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If you want to download or read this book, copy link or url below in the New tab ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ,Download or read Ebooks here ... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ,Download or read Ebooks here ... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[2A4]DeepLearningAtNAVER

  1. 1. Deep Learning at NAVER DEVIEW 2014 NAVER LABS 김 정 희 수석연구원 2014. 09. 30
  2. 2. Contents 2 1. Deep learning overview 2. Deep learning at NAVER 3. Deep learning?
  3. 3. 1. Deep Learning Overview
  4. 4. Deep Learning Overview 4 Machine learning with Deep neural networks
  5. 5. Machine Learning 5 Observable Data Modeling (Learning) Informa?on
  6. 6. Machine Learning 6 Observable Data Modeling (Learning) Informa?on Speech DEVIEW 2014 Recogni?on Speech Text
  7. 7. Machine Learning 7 Observable Data Modeling (Learning) Informa?on One Language Another Language Machine I am a boy Transla?on 나는 소년이다
  8. 8. Machine Learning 8 Observable Data Modeling (Learning) Informa?on Image 고양이 Classify Image Category
  9. 9. Machine Learning 9 Observable Data Modeling (Learning) Informa?on HOW ?
  10. 10. Machine Learning 10 Observable Data Modeling (Learning) Informa?on LDA
  11. 11. Machine Learning 11 Observable Data Modeling (Learning) Informa?on Gaussian Mixture
  12. 12. Machine Learning 12 Observable Data Modeling (Learning) Informa?on Non-­‐nega8ve Matrix Factoriza8on
  13. 13. Machine Learning 13 Observable Data Modeling (Learning) Informa?on Deep Neural Networks
  14. 14. Deep Neural Networks -­‐ Learning 14 Input Data Weight Matrix Probability of Each Class Back propaga8on 1980년대 확립
  15. 15. Why Deep Learning? Yoshua Bengio 2014 KDDNN 15 • NAVER Speech recogniNon § 20 % error rate 개선
  16. 16. Why Deep Learning? Yoshua Bengio 2014 KDDNN 16 • ImageNet ClassificaNon 2012 § Krizhevsky et al (convnet) -­‐ 16.4 % error ( top – 5 ) § Next best ( non – convnet ) -­‐ 26.2 % error
  17. 17. Why Deep Learning? Yoshua Bengio 2014 KDDNN 17
  18. 18. 2. Deep Learning at NAVER
  19. 19. Rebirth of Deep Learning 19 New Algorithms Big Data Hardware
  20. 20. Primary Check Points of Deep Learning 20 New Algorithms Big Data Hardware
  21. 21. ConvoluNonal neural networks 2012 21 FFNN CNN Alex Krizhevsky et al. 2012 NIPSNN
  22. 22. Image classificaNon (m.ndrive.naver.com) 22 FFNN CNN RNN
  23. 23. ConvoluNonal neural networks 2014 23 FFNN CNN Chris?an Szegedy et al. 2014 ILSVRCNN
  24. 24. Deep Neural Networks 24 C_1 C_2 ŸŸŸ C_n FFNN CNN Non Linear Ac?va?on Sigmoid ŸŸŸ ŸŸŸ ŸŸŸ Output Layer Hidden Layer Input Layer
  25. 25. Back -­‐ propagaNon 25 C_1 C_2 ŸŸŸ C_n FFNN CNN Non Linear Ac?va?on Sigmoid ŸŸŸ ŸŸŸ ŸŸŸ Back-­‐ propgata?on Error 의 미분값
  26. 26. Exploding or Vanishing 26 FFNN CNN
  27. 27. Recurrent neural networks 27 FFNN CNN RNN
  28. 28. Recurrent Neural Networks 28 • SequenNal data Analysis § Video § RoboNcs § Language model
  29. 29. Recurrent neural networks 29 FFNN CNN T = R1N0N0 ?
  30. 30. Recurrent Neural Networks 30 100 개의 Hidden Layer
  31. 31. Long Short-­‐Term Memory 31 FFNN CNN T = R1N0N0 ?
  32. 32. Language Model 32 • Language model § w(1), w(2), w(3), … , w(n) 의 단어열이 주어졌을 때, § 확률 p( w(1), w(2), w(3), … , w(m) ) 위한 확률 모델
  33. 33. LM ApplicaNon (1) 33 • Speech recogniNon § 같은 발음 / 다른 철자 o 위성이 지구 ( 궤도 / 괴도 ) 를 돌고 있다. o (궤도 / 괴도) 루팡을 읽었다.
  34. 34. LM ApplicaNon (2) 34 • Machine translaNon § 같은 철자 / 다른 의미 o The town lies on the coat. o She lies about her age.
  35. 35. Recurrent Neural Networks -­‐ LM 35 • 에측 정확도 향상 § 기존 방법 : KN-­‐5 § 실험 Data set : Penn tree bank § 척도 : perplexity § 20% 향상
  36. 36. Primary Check Points of Deep Learning 36 New Algorithms Big Data Hardware
  37. 37. Neural Networks 37 • Data-­‐driven § 학습 Data 에 기초해 해당 Class 의 확률이 최대가 되도록 § Prior knowledge 를 사용하지 않는다. ! 학습 Data를 그대로 모사하는 것
  38. 38. Neural Networks 38 • Big data 가 아니면 문제가 생긴다 § Why? § 보유한 학습 Data 가 이 세상의 모든 Data 라고 할 수 있나?
  39. 39. Before – Big Data 39 Real World Data Small Learning Data
  40. 40. Before – Big Data 40 Real World Data
  41. 41. Before – Big Data 41 Real World Data Small Learning Data
  42. 42. Before – Big Data 42 Real World Data
  43. 43. Before – Big Data 43 Real World Data
  44. 44. Before – Big Data 44 Real World Data ERROR !!!
  45. 45. Human Case 45 MiYe 광고 이미지
  46. 46. Before – Big Data 46 Real World Data ERROR !!!
  47. 47. Prior Know-­‐ ledge Before – Big Data 47 Real World Data Small Leaning Data ERROR !!!
  48. 48. GMM Before – Big Data 48 Real World Data Small Speech Data ERROR !!!
  49. 49. Gabor Filter Before – Big Data 49 Real World Data Small Image Data ERROR !!!
  50. 50. Image classificaNon (m.ndrive.naver.com) 50 FFNN CNN RNN
  51. 51. ConvoluNonal neural networks 51 FFNN CNN Alex Krizhevsky et al. 2012 NIPSNN
  52. 52. 기존 Approach 52 FFNN CNN RNN Image Pixels Hand-­‐ designed Feature Extractor Trainable Classifier Object Class
  53. 53. Deep Learning 53 FFNN CNN RNN Image Pixels Trainable Feature Extractor Trainable Classifier Object Class
  54. 54. ConvoluNonal neural networks 54 FFNN CNN Hierarchical Trained Filters Alex Krizhevsky et al. 2012 NIPSNN
  55. 55. RepresentaNon Learning 55 RNN Rob Fergus 2013 NIPSNN
  56. 56. Prior Know-­‐ ledge Afer – Big Data 56 Real World Data Small Leaning Data ERROR !!! Big Learning Data
  57. 57. Big Data 57 • Neural networks § Supervised learning § 무작정 data 만 많이 있어서는 안된다 § 정답이 있는 data가 많이 있어야 한다
  58. 58. Big Data 58 • 정답이 있는 DB를 대량으로 구축하기 힘들다면… § Semi-­‐supervised learning § Supervised Learning + Unsupervised Learning
  59. 59. Big Data 59 • Big Data 를 구축하기 힘들다면… § 서비스 개발 진입 장벽
  60. 60. Big Data 60 • Big Data 를 구축하기 힘들다면… § 일단 만들고 beta 서비스 하자 !
  61. 61. Transfer Learning 61 • Deep learning ! RepresentaNon learning § 유사 Domain 에서 학습된 내용을 다른 도메인으로 § 한국어 음성인식 ! 일본어 음성인식 § 중국어 OCR ! 일본어 OCR § 이미지 feature extractor
  62. 62. Transfer Learning – Image 62 FFNN CNN Supervised Trained Feature Extractor Alex Krizhevsky et al. 2012 NIPSNN
  63. 63. Transfer Learning – Speech RecogniNon 63
  64. 64. Speech recogniNon 64 FFNN CNN RNN • Transfer learning § 일본어, 영어 인식 엔진 개발 § 소량의 데이터 § 한국어와 유사한 정확도
  65. 65. Primary Check Points of Deep Learning 65 New Algorithms Big Data Hardware
  66. 66. 계산량 7000 3000 … 3000 500 5 hidden layers 58.5 M parameters
  67. 67. Sparseness • 작은 weight 값들 존재
  68. 68. 계산량 감소 Idea 모든 weight가 필요하지는 않다.
  69. 69. Dimension ReducNon • PCA와 같은 Dimension ReducNon § Layer의 output을 Feature § Linear TransformaNon을 이용 § 필요없는 Dimension을 줄이는 ReducNon
  70. 70. 7000 R 3000 Dimension ReducNon • 7000 x 3000 >= 7000 x R + 3000 x R • 2100 >= R
  71. 71. • Weight matrix 를 두 개의 matrix 곱으로 분리하는 matrix factorizaNon • Weight matrix는 low-­‐rank matrix Dimension ReducNon [7000 ×3000] =[7000 × R][R×3000]
  72. 72. Dimension ReducNon • NAVER § Weight 64% 감소 § 성능 저하 없음
  73. 73. GPU • OpNmizaNon 의 어려움 § Deep learning 용 GPU library 들 release § Ex) Nvidia -­‐ CuNet
  74. 74. 3. Deep Learning?
  75. 75. Deep Learning – The One Ring? 75 State of the art by Abstruse Goose
  76. 76. Deep Learning – The One Ring? 76 State of the art by Abstruse Goose
  77. 77. Deep Learning – The One Ring? 77 State of the art by Abstruse Goose
  78. 78. Deep Learning – Current Status 78 Yoshua Bengio KDD 2014
  79. 79. 79

×