The document is an introduction to a Python course on Natural Language Processing (NLP), covering concepts like text manipulation, tokenization, and model evaluation techniques. It includes resources for further study and discusses applications such as text classification and sentiment analysis. Key programming libraries and corpus information are also highlighted, alongside practical examples of manipulating text corpora.

1. WWW.HACKYALE.COM
NLP IN PYTHON

2. MANIPULATING TEXT
WEEK 0
NLP IN PYTHON

3. COURSE
INTRODUCTION

4. NATURAL LANGUAGE PROCESSING
Programs that:
“Understand” human speech or text
Mainly statistical
Natural vs. artificial languages
Programming languages are unambiguous
Human speech, not so much

5. YOU’LL LEARN
Text processing
Tokenization, stemming, built-in text corpora
Brown, Reuters, WordNet, make your own!
Text classification
Naive Bayes inference, n-gram language models, sentence
parsing, etc.
Classify by topic (Reuters) or by sentiment (imdb)

6. YOU’LL LEARN
Evaluating your models
F-scores, MaxEnt, selecting appropriate training and testing
sets
Applications, MOOCs, and books for further study

7. SOURCES
Natural Language Processing with Python, by
Steven Bird, Ewan Klein, and Edward Loper.
O'Reilly Media, 978-0-596-51649-9.
MOOC: Natural Language Processing with Dan
Jurafsky and Christopher Manning (Stanford,
Coursera)

8. APPLICATIONS

9. SPELL CHECKING

10. CREDIT: GROUPMAIL
SPAM DETECTION

11. CREDIT: HUTCH CARPENTER, I’M NOT ACTUALLY A GEEK BLOG
VIA SENTIMENT140 & SENTIMENT ANALYZER
SENTIMENT ANALYSIS

12. MACHINE TRANSLATION

13. INFORMATION EXTRACTION

14. IBM WATSON
CREDIT: CNME ONLINE

15. DIALOG SYSTEMS

16. DIALOG SYSTEMS
CREDIT: ABOIET COUNSELING
& THERAPY GROUP

17. ELIMINATE ALL HUMANS
CREDIT: TERMINATOR FRANCHISE &
OUR FUTURE ROBOT OVERLORDS

18. GETTING
STARTED

19. WHY PYTHON?
String processing methods
Excellent statistical libraries
Pandas, numpy, etc.
Natural Language Toolkit
Created in 2002
> 50 text resources

20. ALTERNATIVES
Java
StanfordNLP, mallet, OpenNLP
Ruby
Treat, open-nlp

21. STRING MANIPULATION
string'='“This'is'a'great'sentence.”'
string'='string.split()'
string[:4]'
=>'[“This”,'“is”,'“a”,'“great”]'
string[;1:]'
=>'“sentence.”'
string[::;1]'
=>'[“sentence.”,'“great”,'“a”,'“is”,'“This”]'

22. FILE I/O
f'='open(‘sample.txt’,'‘r’)'
for'line'in'f:
' line'='line.split()'
' for'word'in'line:
' ' if'word'[;2:]'=='‘ly’:'
' ' ' print'word

23. SET-UP
NLTK:'http://www.nltk.org/install.html'
NLTK'DATA:'
import'nltk'
nltk.download()'
Matplotlib:
$'pip'install'matplotlib'

24. NLTK.DOWNLOAD()

25. WEEK 0
nltk.corpus
nltk.tokenize
nltk.chat

26. WHAT IS A
CORPUS?

27. TEXT CORPORA
Large body of text
Some are general (Brown), others are domain specific
(nps_chat)
Some are tagged (Brown), others are raw
Some have very specific uses (movie_reviews - sentiment
analysis)

28. NLTK BOOK: CORPUS CHART
CREDIT: NLTK BOOK

29. BUILT-IN CORPORA
Sentences:
[“I”, “hate”, “cats”, “.”]
Paragraphs:
[[“I”, “hate”, “cats”, “.”], [“Dogs”, “are”, “worse”, “.”]]
Entire file:
List of paragraphs, which are lists of sentences

30. MANIPULATING
CORPORA

31. BROWN CORPUS
1960’s at Brown University
> 1,000,000 words
15 categories
News, mystery, editorial, humor, etc.
Part-of-speech tagged
Widely cited

32. NAVIGATING CORPORA
from'nltk.corpus'import'brown'
brown.categories()'
len(brown.categories())'#'num'categories'
brown.words(categories=[‘news’,'‘editorial’])'
brown.sents()'
brown.paras()'
brown.fileids()'
brown.words(fileids=[‘cr01’])'

33. TOKENS VS.TYPES
Tokens
All of the words (or whatever you choose as your smallest
particle of meaning)
I really love this song. It’s the best song I have ever heard.
I: 2
song: 2
heard: 1

34. TOKENS VS.TYPES
Types
The vocabulary of your text
Everything gets counted exactly once
I really love this song. It’s the best song I have ever heard.
I: 1
song: 1
heard: 1

35. LEXICAL DIVERSITY
from'nltk.corpus'import'brown'
from'__future__'import'division'
tokens'='brown.words()'
types'='set(brown.words())'
#'lexical'diversity'
len(tokens)/len(types)'
=>'20.714'
'

36. BY GENRE?
#'pick'categories'to'compare'from:'
brown.categories()'
#'calculate'lexical'diversity'of'news'articles'
news'='brown.words(categories=‘news’)'
len(news)'/'len(set(news))'
#'versus'fiction'
fiction'='brown.words(categories=‘fiction’)'
len(fiction)'/'len(set(fiction))'

37. GUTENBERG CORPUS
Curated by the NLTK team
Just 18 books/poetry collections
Also look at nltk.corpus.genesis

38. BY AUTHOR?
from'nltk.corpus'import'gutenberg'
gutenberg.fileids()'
shakespeare'='gutenberg.words(fileids=‘blah’)'
len(shakespeare)'/'len(set(shakespeare))'
#'versus''
len(austen)'/'len(set(austen))'

39. WHAT ARE WE MISSING?
What does the Brown corpus mean by “fiction”?
Is this representative of the question we want to ask?
Is 68,488 words enough?
Can the different categories be compared?
Similarly sized data?
Same genres of writing?

40. INVESTIGATE YOUR CORPORA!
from'nltk.corpus'import'brown'
brown.fileids(categories=‘fiction’)'
brown.abspath(‘ck01’)'
#'poke'around'the'corpora'in'/…/nltk_data/corpora'
#'check'out'the'README,'category'list,'actual'files

41. TEXT OBJECTS
VS. CORPORA
OBJECTS

42. TEXT OBJECT VS. CORPUS OBJECT
You have to convert some_corpus.words() to a text
object using the Text method
Ex:
from nltk.text import Text
text = Text(brown.words())

43. TEXT OBJECT VS. CORPUS OBJECT
#'you'have'to'convert'a'corpus'object'to'a'text''
#'object'to'access'additional'methods'
from'nltk.text'import'Text'
text'='Text(brown.words())'

44. TEXT METHODS
from'nltk.corpus'import'some_corpus'
from'nltk.text'import'Text'
text'='Text(some_corpus.words())'
text.concordance(“word”)'
text.similar(“word”)'
text.common_contexts([“word1”,'“word2”])'
text.count(“word”)'
text.collocations()

45. WEBTEXT CORPUS
from'nltk.corpus'import'webtext'
from'nltk.text'import'Text'
dating'='Text(webtext.words(fileids=‘singles.txt’))'
dating.collocations()

46. NPS_CHAT CORPUS
10,567 forum posts by age
Out of 500,000 collected by the naval postgraduate school
Can we identify online sexual predators?
Or categorize aspects of written communication that vary by
age?

47. INVESTIGATING CORPORA
from'nltk.corpus'import'nps_chat'
nps_chat.sents()'

48. AHHH, DOESN’T WORK
#'if'you'ever'get'stuck,'use'these'techniques'
#'to'find'appropriate'methods'for'a'corpus'
#'gives'you'list'of'methods'
dir(nps_chat)'
#'gives'you'documentation'
help(nltk.corpus)'
help(nltk.corpus.nps_chat)

49. FREQUENCY
DISTRIBUTIONS

50. FREQUENCY DISTRIBUTIONS
from'nltk'import'FreqDist'
from'nltk.text'import'Text'
import'matplotlib'
text'='Text(some_corpus.words())'
fdist'='FreqDist(text)'
=>'dict'of'{‘token1’:count1,'‘token2’:count2,…}'
fdist[‘the’]''
=>'returns'the'#'of'times'‘the’'appears'in'text'

51. FREQUENCY DISTRIBUTIONS
#'top'50'words'
fdst.most_common(50)'
#'frequency'plot'
fdist.plot(25,'cumulative=[True'or'False])'
#'list'of'all'1;count'tokens'
fdist.hapaxes()

52. FREQUENCY DISTRIBUTIONS
#'most'frequent'token'
fdist.max()'
#'decimal'frequency'of'a'given'token'
fdist.freq(‘token’)'
#'total'#'of'samples'
fdist.N()

53. PROBLEM
FREQ. DIST. OF BROWN CORPUS

54. PROBLEM
We’re mostly counting words that don’t tell us
anything about the text
“The” is most frequent
Followed by ‘,’
We need stopwords
Commonly used words that you can remove from a corpus
before processing

55. STOPWORDS
from'nltk.corpus'import'stopwords'
sw'='stopwords.words(‘english’)'
old_brown'='brown.words()'
new'='[w'for'w'in'old_brown'if'w.lower()'not'in'sw]'
#'better,'but'we’re'still'counting'punctuation'
#'sample'code';'brown.py

56. PROBLEM
FREQ. DIST. OF BROWN CORPUS

57. STOPWORDS & PUNCTUATION
from'nltk.corpus'import'stopwords'
sw'='stopwords.words(‘english’)'
old_brown'='brown.words()'
new'='[w'for'w'in'old_brown'if'w.lower()'not'in'sw'
' ' ' ' ' ' ' ' ' ' ' and$w.isalnum()]'

58. STILL PROBLEMS
FREQ. DIST. OF BROWN’S NEWS CORPUS

59. HOMEWORK
Play around with new corpora
What are the frequent words of diff. genres?
Look at webtext.fileids() for a diversity of text files
Find one in your area of interest
movie_reviews, udhr, wordnet, etc.
dir(some_corpus) => how do the methods differ?
some_corpus.abspath() => what’s actually in the corpus?

60. WWW.HACKYALE.COM
NLP IN PYTHON

61. CLASSIFYING TEXT
WEEK 1
NLP IN PYTHON

62. SOURCES
Natural Language Processing with Python, by
Steven Bird, Ewan Klein, and Edward Loper.
O'Reilly Media, 978-0-596-51649-9.
MOOC: Natural Language Processing with Dan
Jurafsky and Christopher Manning (Stanford,
Coursera)

63. REVIEW

64. NLTK BOOK: CORPUS CHART
CREDIT: NLTK BOOK

65. INAUGURAL CORPUS
from
nltk.corpus
import
inaugural
import
matplotlib
from
nltk
import
Text
inaug
=
Text(inaugural.words())
inaug.collocations()
inaug.concordance(‘freedom’)
inaug.dispersion_plot([‘freedom’,
‘government’,
‘liberty’,
‘hope’])

66. WHAT IS TEXT
CLASSIFICATION?

67. TEXT CLASSIFICATION
Building a model on some features of a language
Assigning categories to unseen documents, given the
ones you’ve seen
Documents can mean anything:
Emails, news articles, tweets, etc.
Spam detection, article relevance, sentiment analysis, etc.

68. TEXT CLASSIFICATION
On different levels
By word, phoneme, author, genre
Identifying patterns and making predictions

69. TRAINING
Supervised
Hand-labeled documents (i.e. movie reviews with pos. or neg.)
Unsupervised
Completely unlabeled
Semi-supervised
Mixture of both

70. TRAINING
Supervised
Hand-labeled documents (i.e. movie reviews with pos. or neg.)
Unsupervised
Completely unlabeled
Semi-supervised
Mixture of both

71. BAYES RULE

72. NOISY CHANNEL MODEL
There is some process that introduces noise to an
underlying “true” form
Misspelling:
The “true” spelling of a word goes through some noisy process
that produces a misspelled word
Edit distance, position in word, proximity of keys on keyboard,
homophones, confusion of left and right fingers

73. BAYES RULE
cat = category
obs = observation

74. Speech Analysis:
Spelling Corrector:
NLP EXAMPLES

75. DEFINITIONS
P(“keep”|/keep/) - Noisy process
P(/keep/) - Prior probability of the lexicon

76. NAIVE BAYES INFERENCE
Independence assumptions
Noisy model can be interpreted without context
Our case:
Labeled categories
Look at inputs separately
Independence assumption applies to words in the topics

77. LANGUAGE
MODELS

78. LANGUAGE MODEL
N-gram
Simplest and “dumbest” model
Just counts words and their physical relation to each other in
the document
Or phonemes, or letters, etc.
Bag of words
Google n-grams example

79. N-GRAMS
Applications
Author identification
Federalist Papers
Genre identification

80. N-GRAMS
Unigram (simple count):
“This movie is great!”
{‘this’: 1, ‘movie’: 1, …}
Bigram:
“# This movie is great!”
[(‘#’, ‘this’), (‘this’, ‘movie’), (‘movie’, ‘is’), …]

81. N-GRAMS
Trigram:
“# # This movie is great”
[(‘#’, ‘#’, ‘this’), (‘#’, ‘this’, ‘movie’), …]
4-grams, 5-grams, etc, etc.
Trigrams are usually good for English

82. N-GRAMS
You have to strike a balance between
Capturing context (large n)
Creating a generalizable model (small n)
Flaws:
Doesn’t model long-distance dependencies
“The movie that I saw with my friends on Wednesday was
great.”

83. NLP EXAMPLE (BIGRAMS)
P(“I
hate
cats”)
=
P(I
|
START)
P(hate
|
I)
P(cats
|
hate)
How
do
we
measure
this?
Relative
Frequency
Estimation
(i.e.,
counting)
P(hate
|
I)
=
count
“I
hate”
/
count
“I”

84. PROBABILITY
OF A SENTENCE

85. INDEPENDENCE ASSUMPTION
Joint prob of a,b = prob of a given b and the prob of b
P(a, b) = P(a | b) P(b)
Markov Chain:
P(a,b,c,d) = P(a | b,c,d) P(b | c,d) P(c | d) P(d)
Independence Assumption (for Bigrams):
P(a,b,c,d) = P(a | b) P(b | c) P(c | d) P(d)

86. N-GRAMS
#
-­‐*-­‐
coding:
utf-­‐8
-­‐*-­‐
from
nltk
import
bigrams,
ConditionalFreqDist
from
nltk.corpus
import
gutenberg
bi
=
bigrams(gutenberg.words(fileids="austen-­‐
emma.txt"))
cfd
=
ConditionalFreqDist(bi)

87. PROBLEMS WITH N-GRAMS
New words will have 0 counts
If your corpus only recognizes this sentence:
“This is great.”
And you get:
“This is amazing.”
P(this | START) P (is | this) P(amazing | is)
The probability of this sentence is 0.

88. SMOOTHING
Re-distributes probability mass from recognizable
words to unrecognized words
Ideally, pulls more mass from the n-grams we’re not
certain about (i.e., those with low counts)
Basic: Laplace smoothing
+1 to every 0 count then normalize
Add-Lambda: use something other than 1!

89. SMOOTHING
More advanced: Good-Turing
Give the probability of the 1-counts to the 0-counts, the 2-
counts to the 1-counts, the 3-counts to the 2-counts, etc., etc.
Then normalize
You have to stop eventually
Sometimes you’ll have categories that don’t exist (i.e., there are
no words counted 18 times).
What do you do for the 17-count words?

90. BASIC TEXT
CLASSIFICATION

91. IMPORTANT STEPS
Define a features function
{‘feature’ : count/bool, ‘feature2’ : count/bool}
Label words in your corpus with their categories
[(category, word), (category, word), (category, word)]
Create a features set
[(list_of_features, category), …]

92. IMPORTANT STEPS
Create a classifier object
Remember that features are just a dict of the
important aspects of some word/document
Jack => {‘last_letter’ : k}

93. NAMES CORPUS
8000 names, separated into male and female
Think of appropriate features
Overfitting is a problem
Especially with a small data set

94. MOVIE REVIEWS
Mess around with corpus.movie_reviews
See sentiment.py
Play around with numbers in NLTK Book example
(i.e., size of test set vs. training set, # of most frequent words
we consider relevant)

95. ACCURACY VS. F-SCORE
True
False
Chosen
True
Positive
False
Positive
Not
Chosen
False
Negative
True
Negative
Accuracy:
How
often
do
we
choose
correctly?
(i.e.
True
/
False)
True
Negative
might
be
HUGE
compared
to
True
Positive,
skewing
the
results.

96. ACCURACY VS. PRECISION & RECALL
True
False
Chosen
True
Positive
False
Positive
Not
Chosen
False
Negative
True
Negative
Precision:
Of
the
things
our
model
selects,
how
many
belong
in
the
category?
True
Positive
/
False
Positive

97. ACCURACY VS. F-SCORE
True
False
Chosen
True
Positive
False
Positive
Not
Chosen
False
Negative
True
Negative
Recall:
Of
the
things
that
belong
to
the
category,
how
many
does
our
model
select?
True
Positive
/
False
Negative

98. BALANCED F1-SCORE
Combines Precision and Recall values: