Download to read offline
![• “great rooms”
• “great hotel”
• “rooms are terrible”
• “hotel is terrible”
Text Processing
JJ NN
JJ NN
NN VB JJ
NN VB JJ
>> nltk.pos_tag(nltk.word_tokenize("hotel is
terrible"))
[('hotel', 'NN'), ('is', 'VBZ'), ('terrible', 'JJ')]](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-45-2048.jpg)
![Minimal Bolerplate Code
class WordCount(luigi.Task):
date = luigi.DateParameter()
def requires(self):
return InputText(date)
def output(self):
return luigi.LocalTarget(’/tmp/%s' % self.date_interval)
def run(self):
count = {}
for f in self.input():
for line in f.open('r'):
for word in line.strip().split():
count[word] = count.get(word, 0) + 1
f = self.output().open('w')
for word, count in six.iteritems(count):
f.write("%st%dn" % (word, count))
f.close()](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-70-2048.jpg)
![class WordCount(luigi.Task):
date = luigi.DateParameter()
def requires(self):
return InputText(date)
def output(self):
return luigi.LocalTarget(’/tmp/%s' % self.date_interval)
def run(self):
count = {}
for f in self.input():
for line in f.open('r'):
for word in line.strip().split():
count[word] = count.get(word, 0) + 1
f = self.output().open('w')
for word, count in six.iteritems(count):
f.write("%st%dn" % (word, count))
f.close()
Task Parameters](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-71-2048.jpg)
![class WordCount(luigi.Task):
date = luigi.DateParameter()
def requires(self):
return InputText(date)
def output(self):
return luigi.LocalTarget(’/tmp/%s' % self.date_interval)
def run(self):
count = {}
for f in self.input():
for line in f.open('r'):
for word in line.strip().split():
count[word] = count.get(word, 0) + 1
f = self.output().open('w')
for word, count in six.iteritems(count):
f.write("%st%dn" % (word, count))
f.close()
Programmatically Defined Dependencies](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-72-2048.jpg)
![class WordCount(luigi.Task):
date = luigi.DateParameter()
def requires(self):
return InputText(date)
def output(self):
return luigi.LocalTarget(’/tmp/%s' % self.date_interval)
def run(self):
count = {}
for f in self.input():
for line in f.open('r'):
for word in line.strip().split():
count[word] = count.get(word, 0) + 1
f = self.output().open('w')
for word, count in six.iteritems(count):
f.write("%st%dn" % (word, count))
f.close()
Each Task produces an ouput](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-73-2048.jpg)
![class WordCount(luigi.Task):
date = luigi.DateParameter()
def requires(self):
return InputText(date)
def output(self):
return luigi.LocalTarget(’/tmp/%s' % self.date_interval)
def run(self):
count = {}
for f in self.input():
for line in f.open('r'):
for word in line.strip().split():
count[word] = count.get(word, 0) + 1
f = self.output().open('w')
for word, count in six.iteritems(count):
f.write("%st%dn" % (word, count))
f.close()
Write Logic in Python](https://image.slidesharecdn.com/bigdatabogota-2016-160414234821/75/Ayudando-a-los-Viajeros-usando-500-millones-de-Resenas-Hoteleras-al-Mes-74-2048.jpg)
Uploaded byBig Data Colombia
Ayudando a los Viajeros usando 500 millones de Reseñas Hoteleras al Mes
This document discusses how TrustYou processes large amounts of hotel review data to provide summaries to travelers. It crawls over 30 million reviews daily across 25 languages. Natural language processing and machine learning techniques are used to analyze the text and provide recommendations. Workflows are managed through Luigi and tasks include crawling, text processing, modeling word embeddings, and powering a sample application. Hadoop and Python are used extensively to handle the large scale processing.
More Related Content
Similar to Ayudando a los Viajeros usando 500 millones de Reseñas Hoteleras al Mes
Ayudando a los Viajeros usando 500 millones de Reseñas Hoteleras al Mes
- 1. Helping Travellers Make BetterHotel Choices 500 Million Times a Month Miguel Cabrera @mfcabrera https://www.flickr.com/photos/18694857@N00/5614701858/
- 2.
- 3. • Neuberliner • Ing.Sistemas e Inf. Universidad Nacional - Med • M.Sc. In Informatics TUM, Hons. Technology Management. • Work for TrustYou as Data (Scientist|Engineer| Juggler)™ • Founder and former organizer of Munich DataGeeks ABOUT ME
- 4.
- 5. • What wedo • Architecture • Technology • Crawling • Textual Processing • Workflow Management and Scale • Sample Application AGENDA
- 6.
- 7. For every hotelon the planet, provide a summary of traveler reviews.
- 15. • Crawling • NaturalLanguage Processing / Semantic Analysis • Record Linkage / Deduplication • Ranking • Recommendation • Classification • Clustering Tasks
- 16.
- 17. Data Flow Crawling Seman-c Analysis Database API Clients • Google • Kayak+ • TY Analytics
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24. 300,000+ daily newreviews
- 25.
- 26.
- 27. • Numpy • NLTK • Scikit-Learn • Pandas • IPython / Jupyter • Scrapy Python
- 28. • Hadoop Streaming • MRJob • Oozie • Luigi • … Python + Hadoop
- 29.
- 30.
- 35. • Build yourown web crawlers • Extract data via CSS selectors, XPath, regexes, etc. • Handles queuing, request parallelism, cookies, throttling … • Comprehensive and well-designed • Commercial support by http://scrapinghub.com/
- 40. • 2 -3 million new reviews/week • Customers want alerts 8 - 24h after review publication! • Smart crawl frequency & depth, but still high overhead • Pools of constantly refreshed EC2 proxy IPs • Direct API connections with many sites Crawling at TrustYou
- 41. • Custom frameworkvery similar to scrapy • Runs on Hadoop cluster (100 nodes) • Not 100% suitable for MapReduce • Nodes mostly waiting • Coordination/messaging between nodes required: – Distributed queue – Rate Limiting Crawling at TrustYou
- 42.
- 43. Text Processing Raw text Setence spli:ng Tokenizing Stopwords Stemming Topic Models Word Vectors Classification
- 44.
- 45. • “great rooms” • “great hotel” • “rooms are terrible” • “hotel is terrible” Text Processing JJ NN JJ NN NN VB JJ NN VB JJ >> nltk.pos_tag(nltk.word_tokenize("hotel is terrible")) [('hotel', 'NN'), ('is', 'VBZ'), ('terrible', 'JJ')]
- 46. • 25+ languages • Linguistic system (morphology, taggers, grammars, parsers …) • Hadoop: Scale out CPU • ~1B opinions in the database • Python for ML & NLP libraries Semantic Analysis
- 47.
- 48.
- 49. An instance ofshallow learning
- 50.
- 51.
- 52. “king” – “man”+ “woman” = “queen”
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59. Similar words/documents arenearby vectors
- 60. Wor2vec offer asimilarity metric of words
- 61. Can be extendedto paragraphs and documents
- 62. A fast Pythonbased implementation available via Gensim
- 64.
- 65. Crawl Extract Clean Stats ML ML NLP
- 66. Luigi “ A pythonframework for data flow definition and execution ”
- 67. Luigi • Build complexpipelines of batch jobs • Dependency resolution • Parallelism • Resume failed jobs • Some support for Hadoop
- 68.
- 69. Luigi • Dependency definition • Hadoop / HDFS Integration • Object oriented abstraction • Parallelism • Resume failed jobs • Visualization of pipelines • Command line integration
- 70. Minimal Bolerplate Code classWordCount(luigi.Task): date = luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%st%dn" % (word, count)) f.close()
- 71. class WordCount(luigi.Task): date =luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%st%dn" % (word, count)) f.close() Task Parameters
- 72. class WordCount(luigi.Task): date =luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%st%dn" % (word, count)) f.close() Programmatically Defined Dependencies
- 73. class WordCount(luigi.Task): date =luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%st%dn" % (word, count)) f.close() Each Task produces an ouput
- 74. class WordCount(luigi.Task): date =luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%st%dn" % (word, count)) f.close() Write Logic in Python
- 75.
- 77.
- 78. Hadoop Streaming cat input.txt |./map.py | sort | ./reduce.py > output.txt
- 79. Hadoop Streaming hadoop jar contrib/streaming/hadoop-*streaming*.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/text.txt -output /user/hduser/gutenberg-output
- 80. class WordCount(luigi.hadoop.JobTask): date =luigi.DateParameter() def requires(self): return InputText(date) def output(self): return luigi.hdfs.HdfsTarget(’%s' % self.date_interval) def mapper(self, line): for word in line.strip().split(): yield word, 1 def reducer(self, key, values): yield key, sum(values) Luigi + Hadoop/HDFS
- 81.
- 82.
- 83.
- 84. Before • Bash scripts+ Cron • Manual cleanup • Manual failure recovery • Hard(er) to debug
- 85. Now • Complex nestedLuigi jobs graphs • Automatic retries • Still Hard to debug
- 86. We use itfor… • Standalone executables • Dump data from databases • General Hadoop Streaming • Bash Scripts / MRJob • Pig* Scripts
- 87. You can wrapanything
- 88.
- 89.
- 92.
- 93. Reviews highlight theindividuality and personality of users
- 94. Snippets from Reviews “Hipsdon’t lie” “Maid was banging” “Beautiful bowl flowers” “Irish dance, I love that” “No ghost sighting” “One ghost touching” “Too much cardio, not enough squats in the gym” “it is like hugging a bony super model”
- 95. Hotel Reviews +Gensim + Python + Luigi = ?
- 96.
- 98. from gensim.models.doc2vec importDoc2Vec class LearnModelTask(luigi.Task): # Parameters.... blah blah blah def output(self): return luigi.LocalTarget(os.path.join(self.output_directory, self.model_out)) def requires(self): return LearnBigramsTask() def run(self): sentences = LabeledClusterIDSentence(self.input().path) model = Doc2Vec(sentences=sentences, size=int(self.size), dm=int(self.distmem), negative=int(self.negative), workers=int(self.workers), window=int(self.window), min_count=int(self.min_count), train_words=True) model.save(self.output().path)
- 99. Wor2vec/Doc2vec offer asimilarity metric of words
- 100. Similarities are usefulfor non- personalized recommender systems
- 101. Non-personalized recommenders recommend itemsbased on what other consumers have said about the items.
- 102.
- 103.
- 104. Takeaways • It ispossible to use Python as the primary language for doing large data processing on Hadoop. • It is not a perfect setup but works well most of the time. • Keep your ecosystem open to other technologies.
- 105.
- 106.
- 107.