Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep learning - Conceptual understanding and applications

65,555 views

Published on

딥러닝 Deep Learning에 관해서 팀에 공유했던 발표자료입니다.

Published in: Data & Analytics
  • Have u ever tried external professional writing services like ⇒ www.HelpWriting.net ⇐ ? I did and I am more than satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Did u try to use external powers for studying? Like ⇒ www.WritePaper.info ⇐ ? They helped me a lot once.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Deep learning - Conceptual understanding and applications

  1. 1. Deep Learning Conceptual Understanding & Application Variants ben.hur@daumkakao.com The aim of this slide is at conceptual understanding of deep learning, rather than implementing it for practical applications. That is, this is more for paper readers, less for system developers. Note that, some variables are not uniquely defined nor consistently used for convenience. Also, variable/constant and vector/matrix are not clearly distinct. More importantly, some descriptions may be incorrect due to lack of my ability.
  2. 2. Deep Learning (DL) is, in concise, a deep and wide neural network. That is half-true.
  3. 3. Deep Learning is difficult because of lack of understanding about — artificial neural network (ANN) — unfamiliar applications.
  4. 4. DL — ANN — Signal transmission between neurons — Graphical model (Belief network) — Linear regression / logistic regression — Weight, bias & activation — (Feed-forward) Multi-layer perceptron (MLP) — Optimization — Back-propagation — (Stochastic) Gradient-decent
  5. 5. DL — Applications — Speech recognition (Audio) — Natural language processing (Text) — Information retrieval (Text) — Image recognition (Vision) — & yours (+)
  6. 6. Artificial Neural Network (ANN)
  7. 7. ANN is a nested ensemble of [logistic] regressions.
  8. 8. Perceptron & Message Passing
  9. 9. http://en.wikipedia.org/wiki/Neuron
  10. 10. 5 5 y = x Message passing
  11. 11. y = Σ x y = Σ wx y = Σ wx + b Linear regression
  12. 12. y = sigm(Σwx + b) Logistic regression Σwx?Σwx
  13. 13. Activation function — Linear g(a) = a — Sigmoid g(a) = sigm(a) = 1 / (1 + exp(-a)) — Tanh g(a) = tanh(a) = (exp(a) - exp(-a)) / (exp(a) + exp(-a)) — Rectified linear g(a) = reclin(a) = max(0, a) — Step — Gaussian — Softmax, usually for the output layer
  14. 14. Bias controls activation. y = ax b y = ax + b
  15. 15. Multi-Layer Perceptron (MLP)
  16. 16. Linearly non-separable by a single perceptron
  17. 17. Input layer Hidden layer Output layer http://info.usherbrooke.ca/hlarochelle/ neural_networks/content.html Interconnecting weight Bias Activation function
  18. 18. http://info.usherbrooke.ca/hlarochelle/ neural_networks/content.html
  19. 19. Back-Propagation & Optimization
  20. 20. {x, y} h=a(wx+b) y'=a2(w2h+b2) l = y - y'w2 & b2w & b w2 R & b2 RwR & bR y' Back-propagation {x, y} —> feed-forward <— back-propagation
  21. 21. Reinforced through epoch ANN is a reinforcement learning.
  22. 22. Local minimum Plateaus http://info.usherbrooke.ca/hlarochelle/ neural_networks/content.html
  23. 23. Stochastic Gradient Descent The slide excludes details. See reference tutorial for that.
  24. 24. ANN is a good approximator of any functions, if well-designed and trained.
  25. 25. Deep Learning
  26. 26. DL (DNN) is a deep and wide neural network. many (2+) hidden layers many input/hidden nodes
  27. 27. DEEP WIDE Large matrix Many matrix Image from googling
  28. 28. Brain image from Simon Thorpe Other from googling
  29. 29. Many large connection matrices (W) — Computational burden (GPU) — Insufficient labeled data (Big Data) — Under-fitting (Optimization) — Over-fitting (Regularization)
  30. 30. Unsupervised makes DL success.
  31. 31. Unsupervised pre-training — Initialize W matrices in an unsupervised manner — Restricted Boltzmann Machine (RBM) — Auto-encoder — Pre-train in a layer-wise (stacking) fashion — Fine-tune in a supervised manner
  32. 32. Unsupervised representation learning — Construct a feature space — DBM / DBN — Auto-encoder — Sparse coding — Feed new features to a shallow ANN as input cf, PCA + regression
  33. 33. visible hidden Restricted Boltzmann Machine (RBM)
  34. 34. Boltzmann Machine Example
  35. 35. visible hidden
  36. 36. Auto-encoder x x' W WT encoder decoder
  37. 37. if k <= n (single & linear), then AE becomes PCA. otherwise, AE can be arbitrary (over-complete). Hidden & Output layers plays the role of semantic feature space in general - compresses & latent feature (under-complete) - expanded & primitive feature (over-complete)
  38. 38. Denoising auto-encoder x x' nois(x)
  39. 39. Contractive auto-encoder Regularized auto-encoder Same as Ridge (or similar to Lasso) Regression
  40. 40. Hinton and Salakhutdinov, Science, 2006 Deep auto-encoder
  41. 41. (k > n, over-complete) Sparse coding
  42. 42. X Y
  43. 43. RBM3 RBM2 RBM1 Stacking Training in the order of RBM1, RBM2, RBM3, …
  44. 44. Representation Learning Artificial Neural Network+ - Sparse coding - Auto-encoder - DBN / DBM
  45. 45. Stochastic Dropout Training for preventing over-fitting
  46. 46. http://info.usherbrooke.ca/hlarochelle/ neural_networks/content.html
  47. 47. Deep Architecture
  48. 48. Deep architecture is an ensemble of shallow networks.
  49. 49. Key Deep Architectures — Deep Neural Network (DNN) — Deep Belief Network (DBN) — Deep Boltzmann Machine (DBM) — Recurrent Neural Network (RNN) — Convolution Neural Network (CNN) — Multi-modal/multi-tasking — Deep Stacking Network (DSN)
  50. 50. Applications
  51. 51. Applications — Speech recognition — Natural language processing — Image recognition — Information retrieval Application Characteristic - Non-numeric data - Large dimension (curse of dimensionality)
  52. 52. Natural language processing — Word embedding — Language modeling — Machine translation — Part-of-speech (POS) tagging — Named entity recognition — Sentiment analysis — Paraphrase detection — …
  53. 53. Deep-structured semantic modeling (DSSM) BM25(Q, D) vs Corr(sim(Q, D), CTR)
  54. 54. DBN & DBM
  55. 55. Recurrent Neural Network (RNN)
  56. 56. Recurrent Neural Network (RNN)
  57. 57. http://nlp.stanford.edu/courses/NAACL2013/ NAACL2013-Socher-Manning-DeepLearning.pdf Recursive Neural Network
  58. 58. Convolution Neural Network (CNN)
  59. 59. C-DSSM
  60. 60. CBOW & Skip-gram (Word2Vec)
  61. 61. Deep Stacking Network (DSN)
  62. 62. Tensor Deep Stacking Network (TDSN)
  63. 63. Multi-modal / Mulit-tasking
  64. 64. Google Picture to Text
  65. 65. Content-based Spotify Recommendation
  66. 66. In various applications, Deep Learning provides either — better performance than state-of-the-art — similar performance with less labor
  67. 67. Remarks
  68. 68. Deep Personalization with Rich Profile in a Vector Representation - Versatile / flexible input - Dimension reduction - Same feature space ==> Similarity computation
  69. 69. RNN for CF http://erikbern.com/?p=589
  70. 70. Hard work — Application-specific configuration — Network structure — Objective and loss function — Parameter optimization
  71. 71. Yann LeCun —> Facebook Geoffrey Hinton —> Google Andrew Ng —> Baidu Joshua Benjio —> U of Montreal
  72. 72. References Tutorial — http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html — http://deeplearning.stanford.edu/tutorial Paper — Deep learning: Method and Applications (Deng & Yu, 2013) — Representation learning: A review and new perspective (Benjio et al., 2014) — Learning deep architecture for AI (Benjio, 2009) Slide & Video — http://www.cs.toronto.edu/~fleet/courses/cifarSchool09/slidesBengio.pdf — http://nlp.stanford.edu/courses/NAACL2013 Book — Deep learning (Benjio et al. 2015) http://www.iro.umontreal.ca/~bengioy/dlbook/
  73. 73. Open Sources (from wikipedia) — Torch (Lua) https://github.com/torch/torch7 — Theano (Python) https://github.com/Theano/Theano/ — Deeplearning4j (word2vec for Java) https://github.com/SkymindIO/deeplearning4j — ND4J (Java) http://nd4j.org https://github.com/SkymindIO/nd4j — DeepLearn Toolbox (matlab) https://github.com/rasmusbergpalm/DeepLearnToolbox/ graphs/contributors — convnetjs (javascript) https://github.com/karpathy/convnetjs — Gensim (word2vec for Python) https://github.com/piskvorky/gensim — Caffe (image) http://caffe.berkeleyvision.org — More+++
  74. 74. FBI WARNING Some reference citations are missed. Many of them come from the first tutorial and the first paper, and others from googling. All rights of those images captured, albeit not explicitly cited, are reserved to original authors. Free to share, but cautious of that.
  75. 75. Left to Blank

×