Downloaded 13 times
![Word Tokenization!
Splitting a sentence into single words
>>> from nltk.tokenize import word_tokenize
!
>>> word_tokenize("All your base are belong to us")
['All', 'your', 'base', 'are', 'belong', 'to', 'us']](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-15-2048.jpg)
![Sentence Tokenization!
Splitting a text into sentences
>>> from nltk.tokenize import sent_tokenize
!
>>> sent_tokenize("Hello, Mr. Anderson. We missed you!")
['Hello, Mr. Anderson.', 'We missed you!']](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-16-2048.jpg)
![Stemming!
Finding the word stem or root form
>>> import nltk
>>> porter = nltk.PorterStemmer()
>>> lancaster = nltk.LancasterStemmer()
>>> wnl = nltk.WordNetLemmatizer()
!
>>> [wnl.lemmatize(w) for w in ['investigation','women']]
['investigation', ‘woman']
!
>>> [porter.stem(w) for w in ['investigation','women']]
['investig', 'women']
!
>>> [lancaster.stem(w) for w in ['investigation','women']]
['investig', 'wom']](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-18-2048.jpg)
![Part-of-Speech Tagging!
Identifying nouns, verbs, adjectives…
>>> import nltk
>>> text = "In the middle ages Sweden had the
same king as Denmark and Norway."
>>> words = nltk.word_tokenize( text )
!
>>> nltk.pos_tag( words )
[('In', 'IN'), ('the', 'DT'), ('middle', 'NN'),
('ages', 'NNS'), ('Sweden', 'NNP'), ('had', 'VBD'),
('the', 'DT'), ('same', 'JJ'), ('king', 'NN'), ('as',
'IN'), ('Denmark', 'NNP'), ('and', 'CC'), ('Norway',
'NNP'), ('.', '.')]
NN* Noun
VB* Verb
JJ* Adjective
RB* Adverb
DT Determiner
IN Preposition](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-19-2048.jpg)
![Named Entity Recognition!
Identifying people, organizations, locations…
>>> import nltk
>>> text = "New York City is the largest city in the
United States."
>>> words = nltk.word_tokenize( text )
!
>>> nltk.ne_chunk( nltk.pos_tag( words ) )
Tree('S', [Tree('GPE', [('New', 'NNP'), ('York', 'NNP'),
('City', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'),
('largest', 'JJS'), ('city', 'NN'), ('in', 'IN'), ('the',
'DT'), Tree('GPE', [('United', 'NNP'), ('States',
'NNPS')]), ('.', '.')])
ORGANIZATION Georgia-Pacific Corp., WHO
PERSON Eddy Bonte, President Obama
LOCATION Murray River, Mount Everest
DATE June, 2008-06-29
TIME two fifty a m, 1:30 p.m.
MONEY GBP 10.40
PERCENT twenty pct, 18.75 %
FACILITY Washington Monument, Stonehenge
GPE South East Asia, Midlothian
(geo-political entity)](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-20-2048.jpg)
![T. Mikolov, K. Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word
Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online].
Available: http://arxiv.org/abs/1301.3781
Vectors can encode relationships
MAN
WOMAN
AUNT
UNCLE
QUEEN
KING
word2vec
Representing a word with a vector](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-29-2048.jpg)
![T. Mikolov, K. Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word
Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online].
Available: http://arxiv.org/abs/1301.3781
man is to woman as king is to ?
KINGS
KING
QUEEN
QUEENS
word2vec
Representing a word with a vector](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-30-2048.jpg)
![T. Mikolov, K. Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word
Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online].
Available: http://arxiv.org/abs/1301.3781
word2vec
Representing a word with a vector](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-31-2048.jpg)
![Machine Translation
T. Mikolov, Q. V. Le, and I. Sutskever, ‘Exploiting Similarities among Languages
for Machine Translation’, CoRR, vol. abs/1309.4168, 2013 [Online]. Available:
http://arxiv.org/abs/1309.4168](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-50-2048.jpg)
![{ "event":[
{"query":
{"id":[
{"timestamp_usec":"1317002730153183"}
],
"query_text":"google hangout"
}
},
{"query":
{"id":[
{"timestamp_usec":"1316577601549660"}
],
"query_text":"eurokrise"
}
},
{"query":
{"id":[
{"timestamp_usec":"1315592145720230"}
],
"query_text":"hoverboard"
}
}
parsed_json[‘event’][42]['query']['query_text']](https://image.slidesharecdn.com/heuerhackinghumanlanguagepycon-150526141914-lva1-app6892/75/Hacking-Human-Language-PyCon-Sweden-2015-53-2048.jpg)
Uploaded byhen_drik
Hacking Human Language (PyCon Sweden 2015)
The document discusses the intersection of computational social science and natural language processing, emphasizing the application of various techniques like word vector representations, sentiment analysis, and machine translation. It highlights numerous studies analyzing digital texts and news content to reveal patterns in readability, gender bias, and linguistic subjectivity. Additionally, it provides practical examples of natural language processing methods, including tokenization, stemming, and named entity recognition.
More Related Content
Similar to Hacking Human Language (PyCon Sweden 2015)
Hacking Human Language (PyCon Sweden 2015)
- 1.
- 2.
- 3. – Hacker Ethics “Accessto computers — and anything which might ! teach you something about ! the way the world works! — should be unlimited and total. Always yield to ! the Hands-On Imperative!” Levy, Steven (2001). Hackers: Heroes of the Computer Revolution (updated ed.). New York: Penguin Books. ISBN 0141000511. OCLC 47216793.
- 4. Agenda • Computational SocialScience • Natural Language Processing • Word Vector Representations • Visualising and comparing my Google searches
- 5. D. Crandall andN. Snavely, ‘Modeling People and Places with Internet Photo Collections’, Commun. ACM, vol. 55, no. 6, pp. 52–60, Jun. 2012. DOI: 10.1145/2184319.2184336
- 6. Computational Social Science DigitalHumanities • combines computer science & social sciences • makes new research possible, e.g. the analysis of massive social networks and content of millions of books immersion.media.mit.edu
- 7. Massive-scale automated ! analysisof news-content • 2.5 million articles from 498 different English-language news outlets (Reuters & New York Times Corpus) • automatically annotated into 15 topic areas • the topics were compared in regards to readability, linguistic subjectivity and gender imbalances I. Flaounas, O. Ali, T. Lansdall-Welfare, T. De Bie, N. Mosdell, J. Lewis, and N. Cristianini, ‘Research Methods in the Age of Digital Journalism: Massive-scale automated analysis of news-content: topics, style and gender’, Digital Journalism, vol. 1, no. 1, 2013. DOI:10.1080/21670811.2012.714928
- 8. Linguistic Subjectivity! Adjectives (Part-of-SpeechTagging) & SentiWordNet
- 9. “Low level ofpolitical interest and engagement could be connected to the ! lack of subjectivity (adjectival excess)” Linguistic Subjectivity! Adjectives (Part-of-Speech Tagging) & SentiWordNet
- 10. Male-to-Female Ratio! Named EntityRecognition
- 11. Male-to-Female Ratio! Named EntityRecognition “Gender bias in sports coverage (...) females only account for between only 7 and 25 per cent of coverage”
- 12. scikit-learn gensim Natural Language Toolkit spaCyword2vec MachineLearning Text Processing Topic Modeling Visualization d3.js Google Chart API Highcharts
- 13. Introduction to NaturalLanguage Processing
- 14.
- 15. Word Tokenization! Splitting asentence into single words >>> from nltk.tokenize import word_tokenize ! >>> word_tokenize("All your base are belong to us") ['All', 'your', 'base', 'are', 'belong', 'to', 'us']
- 16. Sentence Tokenization! Splitting atext into sentences >>> from nltk.tokenize import sent_tokenize ! >>> sent_tokenize("Hello, Mr. Anderson. We missed you!") ['Hello, Mr. Anderson.', 'We missed you!']
- 17. Sentence Tokenization! Splitting atext into sentences >>> import nltk >>> import functools ! >>> sent_tokenize = nltk.data.load(“tokenizers/punkt/swedish.pickle”)
- 18. Stemming! Finding the wordstem or root form >>> import nltk >>> porter = nltk.PorterStemmer() >>> lancaster = nltk.LancasterStemmer() >>> wnl = nltk.WordNetLemmatizer() ! >>> [wnl.lemmatize(w) for w in ['investigation','women']] ['investigation', ‘woman'] ! >>> [porter.stem(w) for w in ['investigation','women']] ['investig', 'women'] ! >>> [lancaster.stem(w) for w in ['investigation','women']] ['investig', 'wom']
- 19. Part-of-Speech Tagging! Identifying nouns,verbs, adjectives… >>> import nltk >>> text = "In the middle ages Sweden had the same king as Denmark and Norway." >>> words = nltk.word_tokenize( text ) ! >>> nltk.pos_tag( words ) [('In', 'IN'), ('the', 'DT'), ('middle', 'NN'), ('ages', 'NNS'), ('Sweden', 'NNP'), ('had', 'VBD'), ('the', 'DT'), ('same', 'JJ'), ('king', 'NN'), ('as', 'IN'), ('Denmark', 'NNP'), ('and', 'CC'), ('Norway', 'NNP'), ('.', '.')] NN* Noun VB* Verb JJ* Adjective RB* Adverb DT Determiner IN Preposition
- 20. Named Entity Recognition! Identifyingpeople, organizations, locations… >>> import nltk >>> text = "New York City is the largest city in the United States." >>> words = nltk.word_tokenize( text ) ! >>> nltk.ne_chunk( nltk.pos_tag( words ) ) Tree('S', [Tree('GPE', [('New', 'NNP'), ('York', 'NNP'), ('City', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('largest', 'JJS'), ('city', 'NN'), ('in', 'IN'), ('the', 'DT'), Tree('GPE', [('United', 'NNP'), ('States', 'NNPS')]), ('.', '.')]) ORGANIZATION Georgia-Pacific Corp., WHO PERSON Eddy Bonte, President Obama LOCATION Murray River, Mount Everest DATE June, 2008-06-29 TIME two fifty a m, 1:30 p.m. MONEY GBP 10.40 PERCENT twenty pct, 18.75 % FACILITY Washington Monument, Stonehenge GPE South East Asia, Midlothian (geo-political entity)
- 21. Sentiment Analysis Tell ifa sentence is positive or negative
- 22. Stanford Core NLPTools
- 23.
- 24. –J. R. Firth1957 “You shall know a word by the company it keeps” Firth, John R. 1957. A synopsis of linguistic theory 1930–1955. In Studies in linguistic analysis, 1–32. Oxford: Blackwell.
- 25. –J. R. Firth1957 “You shall know a word by the company it keeps” Quoted after Socher
- 27. Vectors are directionsin space
- 28. Vectors are directionsin space Quoted after Socher word2vec Representing a word with a vector
- 29. T. Mikolov, K.Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online]. Available: http://arxiv.org/abs/1301.3781 Vectors can encode relationships MAN WOMAN AUNT UNCLE QUEEN KING word2vec Representing a word with a vector
- 30. T. Mikolov, K.Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online]. Available: http://arxiv.org/abs/1301.3781 man is to woman as king is to ? KINGS KING QUEEN QUEENS word2vec Representing a word with a vector
- 31. T. Mikolov, K.Chen, G. Corrado, and J. Dean, ‘Efficient Estimation of Word Representations in Vector Space’, CoRR, vol. abs/1301.3781, 2013 [Online]. Available: http://arxiv.org/abs/1301.3781 word2vec Representing a word with a vector
- 32.
- 33.
- 34.
- 35.
- 36.
- 38.
- 39.
- 40.
- 41.
- 42. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al
- 43. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al 2 words context window
- 44. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al 5 words context window 2 words context window
- 45. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al
- 46. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al
- 47. spaCy! Dependency-Based Word representations by Levyand Goldberg Gensim! word2vec by Mikolov et al
- 48.
- 49.
- 50. Machine Translation T. Mikolov,Q. V. Le, and I. Sutskever, ‘Exploiting Similarities among Languages for Machine Translation’, CoRR, vol. abs/1309.4168, 2013 [Online]. Available: http://arxiv.org/abs/1309.4168
- 51.
- 52.
- 53.
- 54.
- 55.
- 56. 1. Find Word Representations word2vec 2. Dimensionality Reduction t-SNE 3. Output JSON Link gensim:https://radimrehurek.com/gensim/! Link word2vec: https://code.google.com/p/word2vec/
- 60.
- 61.
- 62.
- 63.
- 64.
- 65. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 66. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 67. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 68. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 69. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 70. My Google Searches Oct– Dec 2014 Jul – Sep 2011 Both, 2011 & 2014
- 71. Hacking! Human! Language! Hendrik Heuer PyCon ! Stockholm! Sweden hendrikheuer@gmail.com! http://hen-drik.de! @hen_drik Thanks toAndrii, Jussi & Roelof Slides: https://tinyurl.com/pycon-word2vec
- 72. predict the currentword! input! wi-2, wi-1, wi+1, wi+2 ! output ! wi!
- 73. predict the currentword! input! wi-2, wi-1, wi+1, wi+2 ! output ! wi! predict the surrounding words! input wi ! output ! wi-2, wi-1, wi +1, wi +2.