Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Intuition & Use-Cases of Embeddings in NLP & beyond

18 views

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2LZgiKO.

Jay Alammar talks about the concept of word embeddings, how they're created, and looks at examples of how these concepts can be carried over to solve problems like content discovery and search ranking in marketplaces and media-consumption services (e.g. movie/music recommendations). Filmed at qconlondon.com.

Jay Alammar is VC and ML Explainer at STVcapital. He has helped tens of thousands of people wrap their heads around complex ML topics. He harnesses a visual, highly-intuitive presentation style to communicate concepts ranging from the most basic intros to data analysis, interactive intros to neural networks, to dissections of state-of-the-art models in Natural Language Processing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Intuition & Use-Cases of Embeddings in NLP & beyond

  1. 1. Embeddings in NLP and Beyond Jay Alammar — QCon 2019 “There is in all things a pattern that is part of our universe. It has symmetry, elegance, and grace - those qualities you find always in that which the true artist captures. You can find it in the turning of the seasons, in the way sand trails along a ridge, in the branch clusters of the creosote bush or the pattern of its leaves. We try to copy these patterns in our lives and our society, seeking the rhythms, the dances, the forms that comfort. Yet, it is possible to see peril in the finding of ultimate perfection. It is clear that the ultimate pattern contains it own fixity. In such perfection, all things move toward death.” ~ Dune (1965)
  2. 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ nlp-word-embedding • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  4. 4. Google Duplex: A.I. Assistant Calls Local Businesses To Make Appointments https://www.youtube.com/watch?v=D5VN56jQMWM 0:45 - 02:14
  5. 5. Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone https://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html
  6. 6. Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone https://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html Voice to text Text to speech
  7. 7. Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone https://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html Voice to text Text to speech Text input to text response
  8. 8. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016) https://arxiv.org/abs/1609.08144
  9. 9. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016) https://arxiv.org/abs/1609.08144 2) Encoder 3) Decoder 1) Input (Embeddings) 4) Output
  10. 10. Google Duplex Google Translate
  11. 11. Google Duplex Google Translate Embeddings
  12. 12. SECTIONS Intro 1. Personality Embedding 2. Word Embedding Properties in NLP 3. Language Modeling 4. Language Modeling Training 5. Skipgram 6. Negative sampling 7. Sexist embeddings? Non-NLP 8. Airbnb product encoding 9. Alibaba Recommendations 10. ASOS Lifetime Value 11. Other applications 12. resources 13. Consequences
  13. 13. Theme Dune (1965 novel) + five sequels
  14. 14. jalammar.github.io Twitter: jalammar The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) The Illustrated Transformer A Gentle Visual Intro to Data Analysis in Python Using Pandas A Visual and Interactive Guide to the Basics of Neural Networks Instructor on Udacity AI programs: • Machine Learning Engineer Nanodegree • Deep Learning Nanodegree • Natural Language Processing Nanodegree
  15. 15. “I give you the desert chameleon, whose ability to blend itself into the background tells you all you need to know about the roots of ecology and the foundations of a personal identity” ~Dune Personality Embedding What are you like?
  16. 16. Myers-Briggs Type Indicator Introversion Extraversion Sensing Intuition Thinking Feeling Judging Percieving
  17. 17. Big Five Personality Traits Openness Conscientiousness Extraversion Agreeableness Neuroticism,
  18. 18. Big Five Personality Traits practical/conventional curious/independent Impulsive, disorganised Hardworking Quiet/reserved Outgoing Critical, suspicious Helpful, empathetic Calm Anxious Openness Conscientiousness Extraversion Agreeableness Neuroticism,
  19. 19. https://projects.fivethirtyeight.com/personality-quiz/
  20. 20. Extraversion Introversion 38 Extraversion 100 0 Jay
  21. 21. Extraversion Introversion -0.4 Extraversion 1 -1 Jay
  22. 22. Extraversion Introversion 1 -1 -0.4 0.8 Trait#1Trait#2 1-1 Jay
  23. 23. Extraversion Introversion 1 -1 -0.4 0.8 Trait#1Trait#2 1-1 Jay
  24. 24. Extraversion Introversion 1 -1 -0.4 0.8 Trait#1Trait#2 1-1 -0.3 0.2 -0.5 -0.4 Person #1 Person #2 Jay
  25. 25. Extraversion Introversion 1 -1 -0.4 0.8 Trait#1Trait#2 1-1 -0.3 0.2 -0.5 -0.4 Person #1 Person #2 Jay Similarity( Similarity( , ) , )
  26. 26. -0.4 0.8 -0.3 0.2 -0.4 0.8 -0.5 -0.4 , , ) ) cosine_similarity( cosine_similarity(
  27. 27. -0.4 0.8 -0.3 0.2 -0.4 0.8 -0.5 -0.4 cosine_similarity( cosine_similarity( , , ) = 0.87 ) = -0.20 ✓
  28. 28. -0.4 0.8 0.5 -0.2 0.3 Trait#1Trait#2 -0.3 0.2 0.3 -0.4 0.9 -0.5 -0.4 -0.2 0.7 -0.1 Person #1 Person #2 Jay Trait#3Trait#4Trait#5
  29. 29. -0.4 0.8 0.5 -0.2 0.3 Trait#1Trait#2 -0.3 0.2 0.3 -0.4 0.9 -0.5 -0.4 -0.2 0.7 -0.1 Person #1 Person #2 Jay cosine_similarity( , ) , )Trait#3Trait#4Trait#5 cosine_similarity(
  30. 30. 1- We can represent things (and people) as vectors of numbers (Which is great for machines!) -0.4 0.8 0.5 -0.2 0.3Jay
  31. 31. 1- We can represent things (and people) as vectors of numbers (Which is great for machines!) -0.4 0.8 0.5 -0.2 0.3Jay 2- We can easily calculate how similar vectors are to each other
  32. 32. 1- We can represent things (and people) as vectors of numbers (Which is great for machines!) -0.4 0.8 0.5 -0.2 0.3Jay 2- We can easily calculate how similar vectors are to each other Customers who liked Jay also liked: Person #1 Person #2 Person #3 0.86 0.5 -0.20 cosine_similarity
  33. 33. “The gift of words is the gift of deception and illusion” ~ Dune Word Embeddings
  34. 34. “You shall know a word by the company it keeps.” John Rupert Firth
  35. 35. “king”
  36. 36. “king” Embedding: GloVe Dimensions: 50 Trained on: Wikipedia + Gigaword 5 (6B tokens) Number of vectors: 400,000
  37. 37. “king”
  38. 38. “king”
  39. 39. “king” “Man” “Woman”
  40. 40. queen woman girl boy man king queen water
  41. 41. Analogies
  42. 42. king - man + woman ~=
  43. 43. king - man + woman ~= queen
  44. 44. king - man + woman ~= queen king man woman king-man+woman queen
  45. 45. france is to paris as italy is to ______
  46. 46. france is to paris as italy is to ______
  47. 47. “The prophet is not diverted by illusions of past, present and future. The fixity of language determines such linear distinctions. Prophets hold a key to the lock in a language. This is not a mechanical universe. The linear progression of events is imposed by the observer. Cause and effect? That's not it at all. The prophet utters fateful words. You glimpse a thing "destined to occur." But the prophetic instant releases something of infinite portent and power. The universe undergoes a ghostly shift.” ~Dune Language Modeling Early neural language prediction
  48. 48. Thou shalt ______
  49. 49. Trained Model Task: Predict the next word Thou shalt not
  50. 50. Trained Model Task: Predict the next word Prediction 0% aardvark 0% aarhus 0.1% aaron 0% estás … 40% not … 0.01 zyzzyva Thou shalt
  51. 51. 0% aardvark 0% aarhus 0.1% aaron 0% estás … 40% not … 0.01 zyzzyva Thou
 shalt Trained Model Task: Predict the next word 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary Input Features Output Prediction
  52. 52. 0% aardvark 0% aarhus 0.1% aaron 0% estás … 40% not … 0.01 zyzzyva Thou
 shalt Trained Model Task: Predict the next word 1) Look up embeddings aardvark … … shalt … thou … zyzzyva thou
 shalt Input Output Features Prediction
  53. 53. http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
  54. 54. Language Model Training “A process cannot be understood by stopping it. Understanding must move with the flow of the process, must join it and flow with it.” ~Dune
  55. 55. Dataset: Model: Objective: Predict the next word Untrained Model Input 1 Input 2 Output
  56. 56. “Thou shalt not make a machine in the likeness of a human mind” ~Dune
  57. 57. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt not make a machine in the … DatasetSliding window across running text
  58. 58. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt not make a machine in the … DatasetSliding window across running text
  59. 59. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt notthou shalt not make a machine in the … DatasetSliding window across running text
  60. 60. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt notthou shalt not make a machine in the … thou shalt not make a machine in the DatasetSliding window across running text
  61. 61. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt not shalt not make thou shalt not make a machine in the … thou shalt not make a machine in the DatasetSliding window across running text
  62. 62. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt not shalt not make not make a thou shalt not make a machine in the … thou shalt not make a machine in the thou shalt not make a machine in the DatasetSliding window across running text
  63. 63. Thou shalt not make a machine in the likeness of a human mind input 1 input 2 output thou shalt not shalt not make not make a make a machine a machine in thou shalt not make a machine in the … thou shalt not make a machine in the thou shalt not make a machine in the thou shalt not make a machine in the thou shalt not make a machine in the DatasetSliding window across running text
  64. 64. Jay was hit by a ______
  65. 65. Jay was hit by a ______
  66. 66. Jay was hit by a ______bus?
  67. 67. Jay was hit by a ______ bus in…
  68. 68. Jay was hit by a ______ bus in… bus? X red
  69. 69. Jay was hit by a ______ bus in… Jay was hit by a ______
  70. 70. skipgram “Intelligence takes chance with limited d a t a i n a n a r e n a where mistakes are not only possible but a l s o n e c e s s a r y . ” ~Dune
  71. 71. Jay was hit by a ______ bus in…
  72. 72. Jay was hit by a ______ bus in… by a red bus in input 1 input 2 input 3 input 4 output by a bus in red Continuous Bag of Words (CBoW)
  73. 73. Jay was hit by a red bus in…
  74. 74. Jay was hit by a red bus in… by a red bus in
  75. 75. Jay was hit by a red bus in… by a red bus in input output red by red a red bus red in
  76. 76. Thou shalt not make a machine in the likeness of a human mind thou shalt not make a machine in the … input word target word
  77. 77. Thou shalt not make a machine in the likeness of a human mind thou shalt not make a machine in the … input word target word not thou not shalt not make not a
  78. 78. Thou shalt not make a machine in the likeness of a human mind thou shalt not make a machine in the … input word target word not thou not shalt not make not a thou shalt not make a machine in the …
  79. 79. Thou shalt not make a machine in the likeness of a human mind input word target word not thou not shalt not make not a make shalt make not make a make machine thou shalt not make a machine in the … thou shalt not make a machine in the …
  80. 80. Thou shalt not make a machine in the likeness of a human mind input word target word not thou not shalt not make not a make shalt make not make a make machine thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the …
  81. 81. Thou shalt not make a machine in the likeness of a human mind thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the … thou shalt not make a machine in the … input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness
  82. 82. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness
  83. 83. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness
  84. 84. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Model Task: Predict neighbouring word not
  85. 85. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Model Task: Predict neighbouring word not 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary
  86. 86. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Model Task: Predict neighbouring word not 0 aardvark 0 aarhus 0.001 aaron 0 estás … 0.4 taco 0.001 thou … 0.0001 zyzzyva
  87. 87. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Modelnot Model Prediction Actual Target 0 0 0.001 0 … 0.4 0.001 … 0.0001 aardvark aarhus aaron estás taco thou zyzzyva 0 0 0 0 … 0 1 … 0 -
  88. 88. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Modelnot Model Prediction Actual Target 0 0 0.001 0 … 0.4 0.001 … 0.0001 0 0 0 0 … 0 1 … 0 - 0 0 0.001 0 … 0.4 -.0.999 … 0 Error =
  89. 89. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Modelnot Model Prediction Actual Target 0 0 0.001 0 … 0.4 0.001 … 0.0001 0 0 0 0 … 0 1 … 0 - 0 0 0.001 0 … 0.4 -.0.999 … 0 Error = Update Model Parameters
  90. 90. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Modelnot Model Prediction Actual Target - Error = Update Model Parameters
  91. 91. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Modelnot Model Prediction Actual Target - Error = Update Model Parameters
  92. 92. “ T o a t t e m p t a n understanding of Muad'Dib without understanding his m o r t a l e n e m i e s , t h e Harkonnens, is to attempt s e e i n g T r u t h w i t h o u t knowing Falsehood. It is the attempt to see the L i g h t w i t h o u t k n o w i n g Darkness. It cannot be.” ~Dune Negative Sampling
  93. 93. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Model Task: Predict neighbouring word not 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary
  94. 94. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness Untrained Model Task: Predict neighbouring word not 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary [Computationally Intensive]
  95. 95. 1- Generate high quality word embeddings (Don’t worry about next-word prediction) 2- Use these high-quality embeddings to train a language model (to do next-word prediction) We will not worry about this in this talk
  96. 96. Untrained Model Task: Predict neighbouring word not thou Change Task from
  97. 97. Untrained Model Task: Are the two words neighbours? not 0.90 thou To: Untrained Model Task: Predict neighbouring word not thou Change Task from
  98. 98. http://proceedings.mlr.press/v9/gutmann10a/gutmann10a.pdf
  99. 99. input word target word not thou not shalt not make not a make shalt make not make a make machine input word output word target not thou 1 not shalt 1 not make 1 not a 1 make shalt 1 make not 1 make a 1 make machine 1
  100. 100. input word output word target not thou 1 not shalt 1 not make 1 not a 1 make shalt 1 make not 1 make a 1 make machine 1
  101. 101. input word output word target not thou 1 not shalt 1 not make 1 not a 1 make shalt 1 make not 1 make a 1 make machine 1 Smartass Model Task: Are the two words neighbours? def model(in, out): return 1.0 not thou
  102. 102. input word output word target not thou 1 not shalt 1 not make 1 not a 1 make shalt 1 make not 1 make a 1 make machine 1 Smartass Model Task: Are the two words neighbours? def model(in, out): return 1.0 not thou 100% accuracy!!! (but terrible embeddings)
  103. 103. input word output word target not thou 1 not shalt 1 not make 1 not a 1 make shalt 1 make not 1 make a 1 make machine 1
  104. 104. input word output word target not thou 1 not shalt 1 not make 1
  105. 105. input word output word target not thou 1 not 0 not 0 not shalt 1 not make 1 Negative examples
  106. 106. input word output word target not thou 1 not 0 not 0 not shalt 1 not make 1 Negative examples?
  107. 107. input word output word target not thou 1 not 0 not 0 not shalt 1 not make 1 ? Pick randomly from vocabulary (random sampling) aardvark aarhus aaron estás taco thou zyzzyva
  108. 108. input word output word target not thou 1 not aaron 0 not taco 0 not shalt 1 not make 1 Pick randomly from vocabulary (random sampling) aardvark aarhus aaron estás taco thou zyzzyva
  109. 109. input word output word target not thou 1 not aaron 0 not taco 0 not shalt 1 not make 1 Pick randomly from vocabulary (random sampling) Word Count Probability aardvark aarhus aaron estás taco thou zyzzyva
  110. 110. input word output word target not thou 1 not aaron 0 not taco 0 shalt not make a machine Welcome to Word2vec skipgram Negative Sampling
  111. 111. thou shalt not make a machine in the … aardvark … not … shalt … thou … zyzzyva
  112. 112. homepage listing #104 listing #73 listing #3065 Site Search listing #104 listing #73 listing #3065
  113. 113. homepage listing #104 listing #73 listing #3065 Site Search listing #104 listing #73 listing #3065 Listing #1 Listing #2 Listing #3 Listing #4 Listing #5 Listing #6 Listing #7 … Listing #4000
  114. 114. “The machine cannot anticipate every problem of importance to humans. It is the difference between serial bits and an unbroken continuum. We have the one; machines are confined to the other” ~Dune Sexist Embeddings?
  115. 115. france is to paris as italy is to rome
  116. 116. france is to paris as italy is to rome man is to doctor as woman is to ______
  117. 117. man is to doctor as woman is to Nurse
  118. 118. “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” ~ Dune Airbnb Product Embeddings
  119. 119. homepage
  120. 120. homepage listing #104
  121. 121. homepage listing #104 Site Search
  122. 122. homepage listing #104 Site Search listing #73
  123. 123. homepage listing #104 Site Search listing #73 listing #3065
  124. 124. homepage listing #104 listing #73 listing #3065 Site Search listing #104 listing #73 listing #3065
  125. 125. #104 #73 #3065 #100 #200 #300 #400 #500 #600 #700 #100 #200 #300 #400 #500 #600 #700 #800 #900 #1000 #1100 #1200 Click sessions
  126. 126. #100 #200 #300 #200 #500 #600 #700 … Input listing output listing class #200 #100 1 #200 #300 1
  127. 127. #100 #200 #300 #400 #500 #600 #700 … Input listing output listing class #200 #100 1 #200 #300 1 #300 #200 1 #300 #400 1 … … …
  128. 128. #100 #200 #300 #400 #500 #600 #700 … Input listing output listing class #200 #100 1 #200 #417 0 #200 #300 1 #200 #71 0 #300 #200 1 #300 #33 0 #300 #400 1 #200 #22 0 … … …
  129. 129. Listing #1 Listing #2 Listing #3 Listing #4 Listing #5 Listing #6 Listing #7 … Listing #4000 listing #3
  130. 130. Listing #1 Listing #2 Listing #3 Listing #4 Listing #5 Listing #6 Listing #7 … Listing #4000 listing #3 Listing #3 Most similar listings: Listing #72 Listing #2006 Listing #1345 Listing #491 Listing #304
  131. 131. Listing #1 Listing #2 Listing #3 Listing #4 Listing #5 Listing #6 Listing #7 … Listing #4000 listing #3 Similar listings:
  132. 132. Improving Recommendations
  133. 133. listing #3 Listing #3 Most similar listings: Listing #72 Listing #2006 Listing #1345
  134. 134. listing #3 Listing #3 Most similar listings: Listing #72 Listing #2006 Listing #1345 clicked clicked not clicked After recommending:
  135. 135. listing #3 Listing #3 Most similar listings: Listing #72 Listing #2006 Listing #1345 clicked clicked not clicked After recommending: Input listing output listing class #3 #1345 0
  136. 136. Conversion as Global Context
  137. 137. #104 #73 #3065 #100 #200 #300 #400 #500 #600 #700 #100 #200 #300 #400 #500 #600 #700 #800 #900 #1000 #1100 #1200 Click sessions
  138. 138. #100 #200 #300 #200 #500 #600 … #1200 Input listing output listing class #200 #100 1 #200 #300 1 #200 #1200 1 Booked
  139. 139. #100 #200 #300 #400 #500 #600 … #1200 Input listing output listing class #200 #100 1 #200 #300 1 #200 #1200 1 #300 #200 1 #300 #400 1 #300 #1200 1 Booked
  140. 140. https://www.kdd.org/kdd2018/accepted-papers/view/real-time-personalization-using- embeddings-for-search-ranking-at-airbnb
  141. 141. “Prophecy and prescience — How can they be put to the test in the face of the unanswered questions? H o w m u c h [ o f p r o p h e c y a n d prescience] is the prophet shaping the future to fit the prophecy? What of the harmonics inherent in the act of prophecy? Does the prophet see the future or does he see a line of weakness, a fault or cleavage that he may shatter with words or decisions as a diamond-cutter shatters his gem with a blow of a knife?” ~Dune Alibaba Recommendations
  142. 142. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200
  143. 143. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200
  144. 144. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200 500
  145. 145. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200 500400
  146. 146. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200 500400
  147. 147. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200 500400
  148. 148. #100 #200 #500 #100 #400 #100 #400 #500 #100 #300 #500 #200 100 200 500400 300
  149. 149. 100 200 500400 300
  150. 150. 100 200 500400 300 #100
  151. 151. 100 200 500400 300 #100 #400
  152. 152. 100 200 500400 300 #100 #400 #500
  153. 153. 100 200 500400 300 #100 #400 #500 #200
  154. 154. 100 200 500400 300 #100 #400 #500 #200
  155. 155. 100 200 500400 300 #100 #400 #500 #200 #300 #500 #200
  156. 156. 100 200 500400 300 #100 #400 #500 #200 #300 #500 #200 #400 #100 #300 #500 #200 #500
  157. 157. #100 #400 #500 #200 Input item output item class #400 #100 1 #400 #500 1
  158. 158. #100 #400 #500 #200 Input item output item class #400 #100 1 #400 #500 1 #500 #400 1 #500 #200 1
  159. 159. Item #1 Item #2 Item #3 Item #4 Item #5 Item #6 Item #7 … Item #4000
  160. 160. https://www.kdd.org/kdd2018/accepted-papers/view/billion-scale-commodity- embedding-for-e-commerce-recommendation-in-alibaba
  161. 161. “What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking — there's the real danger.” ~ Dune ASOS Customer Lifetime Value Prediction
  162. 162. User #200 User #300 User #500 User #300 User #700 User #400 User #100
  163. 163. Input user output user class #700 #300 1 #700 #400 1 User #300 User #700 User #400 User #100
  164. 164. https://www.kdd.org/kdd2018/accepted-papers/view/perceive-your- users-in-depth-learning-universal-user-representations-from-m
  165. 165. Other Examples: Anghami - Using Word2vec for Music Recommendations https://towardsdatascience.com/using-word2vec-for-music-recommendations- bb9649ac2484 Spotify - Machine learning @ Spotify https://www.slideshare.net/AndySloane/machine-learning-spotify-madison-big-data- meetup
  166. 166. Resources: Speech and Language Processing [Book] Dan Jurafsky and James H. Martin https://web.stanford.edu/~jurafsky/slp3/ Neural Network Methods for Natural Language Processing [Book] Yoav Goldberg https://www.amazon.com/dp/B071FGKZMH/ Chris McCormick [Blog] http://mccormickml.com/
  167. 167. “The thing the ecologically illiterate don’t realize about an ecosystem is that it’s a system. A system! A system maintains a certain fluid stability that can be destroyed by a misstep in just one niche. A system has order, a flowing from point to point. If something dams the flow, order collapses. The untrained might miss that collapse until it was too late. That’s why the highest function of ecology is the understanding of consequences.” ~Dune CONSEQUENCES
  168. 168. "first planetary ecology novel on a grand scale” The Cambridge Companion to Science Fiction "Environmentalists have pointed out that Dune's popularity as a novel depicting a planet as a complex—almost living—thing, in combination with the first images of Earth from space being published in the same time period, strongly influenced environmental movements such as the establishment of the international Earth Day.” Wikipedia
  169. 169. People watch one billion hours of YouTube every day https://youtube.googleblog.com/2017/02/you-know-whats-cool-billion-hours.html
  170. 170. https://qz.com/1178125/youtubes-recommendations-drive-70-of-what-we-watch/
  171. 171. 700,000,000 hours of video / day https://algotransparency.org
  172. 172. Television Invented Years 0 23 46 69 92 92
  173. 173. Television Invented Telephone Invented 0 37.5 75 112.5 150 143 92
  174. 174. Television Invented Telephone Invented Printing Press Invented 0 150 300 450 600 562 143 92
  175. 175. Television Invented Telephone Invented Printing Press Invented Earliest Writing 0 1250 2500 3750 5000 5,000 562 143 92
  176. 176. Television Invented Telephone Invented Printing Press Invented Earliest Writing Agricultural Revolution 0 3000 6000 9000 12000 12,000 5,000 562 143 92
  177. 177. Television Invented Telephone Invented Printing Press Invented Earliest Writing Agricultural Revolution Behavioural Modernity 0 15000 30000 45000 60000 52,000 12,000 5,000 562 143 92
  178. 178. Television Invented Telephone Invented Printing Press Invented Earliest Writing Agricultural Revolution Behavioural Modernity Youtube Recommended Videos Watched / Day 0 20000 40000 60000 80000 79,000 52,000 12,000 5,000 562 143 92
  179. 179. https://www.france24.com/en/20190215-health-science-measles-who-warns-backsliding-cases-soar-worldwide-vaccination-children
  180. 180. https://www.france24.com/en/20190215-health-science-measles-who-warns-backsliding-cases-soar-worldwide-vaccination-children “case numbers worldwide surging around 50 percent last year”
  181. 181. https://www.france24.com/en/20190215-health-science-measles-who-warns-backsliding-cases-soar-worldwide-vaccination-children “case numbers worldwide surging around 50 percent last year” “Last year, measles caused approximately 136,000 deaths around the world”
  182. 182. https://www.france24.com/en/20190215-health-science-measles-who-warns-backsliding-cases-soar-worldwide-vaccination-children “case numbers worldwide surging around 50 percent last year” “Last year, measles caused approximately 136,000 deaths around the world” “experts blame the problem in part on complacency and misinformation about the vaccine […] which have been spread in part on social media”
  183. 183. Hate speechHarmless race discussion racial joke Call for genocide Harmless Harmful
  184. 184. Hate speechHarmless race discussion racial joke Call for genocide PolicyLine Harmless Harmful
  185. 185. Hate speechHarmless race discussion racial joke Call for genocide PolicyLine Harmless Harmful
  186. 186. Hate speechHarmless race discussion racial joke Call for genocide PolicyLine Harmless Harmful
  187. 187. Hate speechHarmless race discussion racial joke Call for genocide Engagement PolicyLine Harmless Harmful
  188. 188. Hate speechHarmless race discussion racial joke Call for genocide PolicyLine Harmless Harmful Engagement
  189. 189. Hate speechHarmless race discussion racial joke Call for genocide PolicyLine Harmless Harmful Engagement
  190. 190. Hate speechHarmless race discussion racial joke Call for genocide Engagement PolicyLine Action: Remove content and/or account
  191. 191. Hate speechHarmless race discussion racial joke Call for genocide Engagement PolicyLine BorderlineContent Action: Remove content and/or account Action: Demote
  192. 192. https://youtube.googleblog.com/2019/01/continuing-our-work-to-improve.html “we’ll begin reducing recommendations of borderline content and content that could misinform users in harmful ways.” “such as: • videos promoting a phony miracle cure for a serious illness, • claiming the earth is flat, or • making blatantly false claims about historic events like 9/11”
  193. 193. “Software is eating the world” ~Marc Andreessen “You can't draw neat lines around planet-wide problems.” ~Dune
  194. 194. https://fullfact.org/
  195. 195. https://fullfact.org/ https://www.bbc.co.uk/news/technology-46836897
  196. 196. https://fullfact.org/ https://www.bbc.co.uk/news/technology-46836897 https://www.youtube.com/watch?v=ddf0lgPCoSo Sentence embeddings for automated factchecking - Lev Konstantinovskiy
  197. 197. https://www.youtube.com/watch?v=ddf0lgPCoSo Sentence embeddings for automated factchecking - Lev Konstantinovskiy Embeddings
  198. 198. “Above all else, the mentat must be a generalist. You can't draw neat lines around planet- wide problems. The generalist looks outward; he looks for living principles, knowing full well that such principles change, that they develop. It is to the characteristics of change itself that the mentat-generalist must look. There can be no permanent catalogue of such change, no handbook or manual. You must look at it with as few preconceptions as possible, asking yourself: "Now what is this thing doing?” ~Dune
  199. 199. BACKUP
  200. 200. input word output word target input • output Softmax Error not thou 1 40 0.95 0.05 not aaron 0 4 0.04 -0.04 not taco 0 2 0.01 -0.01
  201. 201. Deeper dive
  202. 202. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary aardvark … not … shalt … thou … zyzzyva
  203. 203. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not
  204. 204. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not x Hidden layer weights
  205. 205. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not x =
  206. 206. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not x
  207. 207. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not x =
  208. 208. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 1) Look up embeddings 2) Calculate Prediction 3) Project to Output Vocabulary not
  209. 209. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 3) Project to Output Vocabulary 56 900 -70 … aardvark zyzzyva
  210. 210. input word target word not thou not shalt not make not a make shalt make not make a make machine a not makea make a machine a in machine make machine a machine in machine the in a in machine in the in likeness 3) Project to Output Vocabulary 56 900 -70 … aardvark zyzzyva softmax( ) 0.1 0.2 0 … aardvark zyzzyva aarhus
  211. 211. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ nlp-word-embedding

×