19th Athens Big Data Meetup - 2nd Talk - NLP: From news recommendation to word prediction
•
0 likes•42 views
Title: NLP: From news recommendation to word prediction Speaker: Mihalis Papakonstantinou (https://linkedin.com/in/mihalispapak/) Date: Thursday, December 12, 2019 Event: https://meetup.com/Athens-Big-Data/events/266762191/
Recommended
Recommended
More Related Content
Similar to 19th Athens Big Data Meetup - 2nd Talk - NLP: From news recommendation to word prediction
Similar to 19th Athens Big Data Meetup - 2nd Talk - NLP: From news recommendation to word prediction (20)
More from Athens Big Data
More from Athens Big Data (20)
Recently uploaded
Recently uploaded (20)
19th Athens Big Data Meetup - 2nd Talk - NLP: From news recommendation to word prediction
- 1. NLP: from news recommendation to word prediction
- 2. Who we are Vaios Sintsirmas: Technical Manager @AthensVoice vaiossynt@yahoo.gr Mihalis Papakonstantinou: Data Engineer @Agroknow mihalispapak@gmail.com
- 3. FOODAKAI - Data-powered Food Recalls deliverynews.gr - A Text-data Project AthensVoice - A Recommendation Engine TABLE OF CONTENTS NBG’s Word Embedding Challenge 01 02 03 04
- 4. AthensVoice 01 Building a news recommendation engine
- 6. 320.000users/day 510.000pages visited/day 15.000.000new data points/month
- 7. AthensVoice That is just by the website running as-is! What can we do to make it better?
- 8. AthensVoice People are reading article pieces that are interesting to them. So if we present more related articles to them, we can generate even more traffic. And that, just by taking advantage of the traffic generated by the website alone!
- 9. What makes an article interesting? (part#1) “It is interesting to me” I usually read these articles. These articles are from a specific category. Ok, we can present you with more from the same category. “It is interesting to people that have the same interests as me” I have a specific behaviour. But, this matches to the behaviour of others. Ok, we’ll present you with articles that they are reading!
- 10. What makes an article interesting? (part#2) “The article categories I usually read have a high correlation with articles from another category” I am usually reading these categories. Oh wait, these categories have high similarity with these! (based on user data) We’ll present you with articles from these categories! “The articles I usually read are heavily correlated with some other articles” I usually read these articles. Oh wait, the ones you are reading are related with these! (based on text similarity) We’ll present you with these, no problem!
- 11. What makes an article interesting? (part#3) “Usually on this time of day, people are reading articles coming from these categories” It is noon and everyone is getting hungry! So everyone is reading about recipes & restaurants. And we know about it! We’ll show you some of them! “These are trending now!” The social media API(s) are out there. We can get frequently posted hashtags. Can we get related articles? (based on text similarity) Great, we’ll present you with these!
- 13. How did the criteria do?
- 15. In terms of numbers? This is an event sponsored by NBG after all!
- 16. 742.622new page visits/month ~7.800€revenue/month calculating the number of ads served/page and the avg cost of a banner
- 18. Food is recalled every day! All around the world, and for various reasons! Can we collect, process, analyse and present these food recalls?
- 19. 550.000food recalls 1.500hazards/reasons for recalls 17.000ingredients/products involved 47official sources
- 20. Food recall announced in the US Containing a wealth of information!
- 22. We have to start from somewhere! Can we identify the product/ingredient involved? What about the hazard? Let’s employ some vocabulary specific text-mining!
- 24. Company Identification Now we need to identify the company involved in the recall. Let’s employ some NER! But how can we also get the relationship between companies (eg. one is a subsidiary of another)? This sounds like a graph problem!
- 26. Brand Identification Ok, so far so good, but what was the product-brand behind the recall? And more importantly, how can we identify it? There are open-datasets out there! Let’s take advantage of them!
- 28. Misc Info Identification If we also had the date, LOT number & brand size that would be great! Ok, some data source specific parsers should be employed!
- 30. What about?
- 31. Or?
- 32. Or?
- 35. Enough with food recalls! They are scary! (and we are having pizza soon!) Let’s switch to text aggregation!
- 36. deliverynews.gr The Idea And now time for a side-project We know how to handle text data Collect from a variety of sources Employ ML/DL techniques to identify important terms
- 37. deliverynews.gr The Idea Can we put it into practise to collect news-articles announced (currently) within Greece? And identify stuff? And present them? And all of this, fully automated, without manual-labor!
- 38. Let’s try this!
- 42. Articles frequently talk about the same thing Time for some deduplication! Tricky, but can we do something about it?
- 43. Article Deduplication Text similarity matrices are here! But then again, some intuitive rules can be applied! (within period of time & talk about roughly the same thing)
- 44. Article Deduplication Text similarity matrices are here! But then again, some intuitive rules can be applied! (within period of time & talk about roughly the same thing)
- 45. Article Deduplication Text similarity matrices are here! But then again, some intuitive rules can be applied! (within period of time & talk about roughly the same thing) Cosine Similarity on Title: 82% Cosine Similarity on Body: 75%
- 46. What have we collected so far?
- 48. Can we give back to the community?
- 49. Greek articles pretrained vectors Let’s try to create and publish pretrained word & phrases vectors on these greek articles! Work in progress!
- 51. The More Data the Better The RecSys got better the more it was deployed FOODAKAI’s internal workflows got better with more recalls deliverynews.gr is getting bigger and better with more data sources
- 52. More Data == More Raw Data Raw data is (almost) useless We need to (pre)process it!
- 53. We need to test heavily!
- 54. NBG’s Word Embedding Challenge A text-handling challenge 04
- 55. NBG Race Challenge#1 Let’s create a word embedding
- 56. NBG Race Challenge#1 Let’s create a word embedding That can predict the next word
- 57. NBG Race Challenge#1 Let’s create a word embedding That can predict the next word Coming from a specific pool of words
- 58. NBG Race Challenge#1 Let’s create a word embedding That can predict the next word Coming from a specific pool of words From a 10-word sentence
- 59. NBG Race Challenge#1 Let’s create a word embedding That can predict the next word Coming from a specific pool of words From a 10-word sentence Given a 70K dataset of greek law texts
- 61. Lesson#1 More Data is Better
- 62. Let’s crawl for More 4 bash scripts + a Java project ~24 hours later +1.6K new documents ~2% increase in dataset Not much, but still!
- 64. Initial Preprocessing We need to convert our text to a format models can understand
- 65. Initial Preprocessing We need to convert our text to a format models can understand Stemming/Lemmatization seems interesting!
- 66. Initial Preprocessing We need to convert our text to a format models can understand Stemming/Lemmatization seems interesting! (Greek) Stopword removal seems valid.
- 67. Initial Preprocessing We need to convert our text to a format models can understand Stemming/Lemmatization seems interesting! (Greek) Stopword removal seems valid. Let’s begin!
- 68. An Example
- 70. Train#1 Time to train our model!
- 71. Train#1 Time to train our model! Let’s start with word2vec
- 73. -1 points
- 74. Here we go again....
- 76. Preprocessing Ok, we need to create a word embedding to cover specific words
- 77. Preprocessing Ok, we need to create a word embedding to cover specific words And we need to predict the 11th word in a sentence
- 78. Preprocessing Ok, we need to create a word embedding to cover specific words And we need to predict the 11th word in a sentence Let’s preprocess our dataset to create, filtering out words not in our target
- 79. Let’s train our word2vec (again)
- 81. 82.16 points
- 82. Training Ok, preprocessing seems valid
- 83. Training Ok, preprocessing seems valid But word2vec takes forever to train!
- 84. Training Ok, preprocessing seems valid But word2vec takes forever to train! Let’s go with fasttext!
- 86. Training Hm, fasttext performs poorly
- 87. Let’s go with glove
- 88. Training Let’s keep the configuration the same and go with glove
- 90. 92.27 points
- 91. WOW! Let’s stick with glove!
- 92. Glove - Params We need to set the following: window, learning_rate, emb_dimesions, epochs
- 93. Glove - Attempt#1 Let’s keep learning_rate, emb_dim, epochs fixed and play around with the window parameter.
- 94. Glove - Attempt#1 Let’s keep learning_rate, emb_dim, epochs fixed and play around with the window parameter. We need to predict the 11th word, let’s go with window: 11
- 95. 93.16 points
- 96. Glove - Attempt#2 What if we increase the window? Let’s go with: 22
- 97. 92.11 points
- 98. Glove - Attempt#3 Hm, lower window seems to perform better Let’s try a window of 6
- 99. 94.8 points
- 100. Glove - Attempt#4 Ok, even better! Let’s lower the window now even more, let’s try window: 1
- 101. 95.1 points
- 102. Enough with the window! Let’s play around with emb_dim
- 103. Glove - Attempt#5 Our initial tests were with emb_dim: 200 Let’s increase this, let’s go with 500
- 104. 95.8 points
- 105. Glove - Attempt#6 Seems to do better! What about 800?
- 106. 95.85 points
- 107. Glove - Attempt#7 Hm.. the more we increase the value, the higher results we get! Let’s go with 1500 and we’ll win!
- 109. Glove - Attempt#8 Ok, ok, let’s try 900 now!
- 110. 96 points
- 111. Great! Now time to experiment on epochs!
- 112. Glove - Attempt#{9, 10, 11} Let’s go with: 200
- 113. Glove - Attempt#{9, 10, 11} Let’s go with: 200
- 114. Glove - Attempt#{9, 10, 11} Let’s go with: 200 250
- 115. Glove - Attempt#{9, 10, 11} Let’s go with: 200 250
- 116. Glove - Attempt#{9, 10, 11} Let’s go with: 200 250 300
- 117. Glove - Attempt#{9, 10, 11} Let’s go with: 200 250 300
- 118. We have a winner!
- 119. But does the Challenge agree?
- 120. 98.17 points Yes! NBG’s leaderboard agrees!