How to Measure Document Similarity and Build Text Classifiers: A First Look at Term Frequency-Inverse Document Frequency (TF-IDF) Representations Text data is potentially valuable for many data science projects but working with text is different from working with structured data. One representation of text that has worked well for many text mining and machine learning applications is the term frequency - inverse document frequency (TF-IDF) vector. In spite of the long winded name, this method is easy to understand, performs well in many applications, and has been implemented in commonly used data science tools. This presentation will introduce TF-IDF and show examples of how to use TF-IDF for document classification and measuring the similarity between documents. This presentation does not assume any background in text mining or natural language processing. Examples will use Python.

1. A First Look at TF-IDF
Dan Sullivan
PP
Portland Data Science Group
March 2, 2017
Portland Data Science Meetup
March 2, 2017

2. What do we do with this?

3. Challenges
No obvious structure
Fully understanding language is hard
Large number of documents
Want to
Find documents based on similarity
Classify documents

4. Fortunately ...
Measuring similarity and
Classifying documents
Does not require fully understanding text

5. Counting Words

6. “PDX Data Science is all about data.”
about all data is pdx science
1 1 2 1 1 1

7. Corpus to Vectors
{
WordsCount
(Term Frequency)

8. Improvement 1: Remove Stop Words
{
WordsCount
Stop
Words

9. Improvement 2: N-grams
{
WordsCount
“computer” ,
“science”
“Computer science”

10. Example: Corpus of Machine Learning Papers
Some terms appear frequently
“Feature”
“Algorithm”
“Training”
Some less frequently
“Reinforcement”
“Non-linear”
“Convolution”

11. Intuition
Combination of words are good indicators of topic of document
Self-driving cars: “automobile”, “driver”, “radar”, “image”, “sensor”
Text mining: “corpus”, “term vector”, “syntax”
Social Network: “graph”, “communities”, “users”, “influence”

12. Intuition
Combination of words are good indicators of topic of document
Self-driving cars: “automobile”, “driver”, “radar”, “image”, “sensor”
Text mining: “corpus”, “term vector”, “syntax”
Social Network: “graph”, “communities”, “users”, “influence”
Words that appear frequently across documents in a corpus are not good
indicators of topic

13. Intuition
Combination of words are good indicators of topic of document
Self-driving cars: “automobile”, “driver”, “radar”, “image”, “sensor”
Text mining: “corpus”, “term vector”, “syntax”
Social Network: “graph”, “communities”, “users”, “influence”
Words that appear frequently across documents in a corpus are not good indicators of topic
Words that appear frequently only within documents about a single topic are good indicators
of topic

14. Formalizing Intuition: TF-IDF
Notation
t - a term
d - a document
D - a set of documents or corpus
N - number of documents in corpus

15. Formalizing Intuition: TF-IDF
Notation
t - a term
d - a document
D - a set of documents or corpus
N - number of documents in corpus
TF - term frequency
tf(t,d) is the number of times a term t occurs in document d

16. Formalizing Intuition: TF-IDF
Notation
t - a term
d - a document
D - a set of documents or corpus
N - number of documents in corpus
TF - term frequency
tf(t,d) is the number of times a term t occurs in document d
IDF - inverse document frequency
idf(t,D) = log(N / | {d in D: t in d} | )

17. Formalizing Intuition: TF-IDF
Notation
t - a term
d - a document
D - a set of documents or corpus
N - number of documents in corpus
TF - term frequency
tf(t,d) is the number of times a term t occurs in document d
IDF - inverse document frequency
idf(t,D) = log(N / | {d in D: t in d} | )
TF-IDF = tf(t,d) * idf(t,D)

18. TF-IDF is
Large when:
There is a large count of a term in a
document (large TF) and ...
Low number of documents with term in
them
Small when
Term appears in many documents in
the corpus
TF-IDF
Frequency
Stop Words
Common Words
Rare Words

19. Improvement 3: TF-IDF
{
WordsTF-IDF

20. Populating a Term Vector with TF-IDF
V[index(“emmy”)] = tf-idf(“emmy”,d)
V[index(“noether”)] = tf-idf(“noether”,d)
V[index(“known”)] = tf-idf(“known”,d)
V[index(“landmark”)] = tf-idf(“landmark”,d)
V[index(“contribution”)] = tf-idf(“contribution”,d)
V[index(“abstract”)] = tf-idf(“abstract”,d)
V[index(“algebra”)] = tf-idf(“algebra”,d)
V[index(“theoretical”)] = tf-idf(“theoretical”,d)
V[index(“physics”)] = tf-idf(“physics”,d)
“Emmy Noether is known for her landmark contributions to
abstract algebra and theoretical physics”

21. Vector Space Model
Term 3
Term 2
Term 1
Doc 1
Doc 2
Doc 3
Term 1 Term 2 Term 3
Doc 1 0.4 0.1 0.6
Doc 2 0.3 0.5 0.0
Doc 3 0.0 0.2 0.6

22. Similarity Measures
Term 3
Term 2
Term 1
Doc 1
Doc 2
Doc 3 Euclidian Distance
Cosine

23. Classify by Vector (Point)
TF-IDF Vector

24. Text Classifier with Scikit Learn

25. Document Similarity with Gensim

26. NLP Tools
Python
Gensim
NLTK
spaCy & textacy
Scikit-Learn
TextBlob
R
TM
OpenNLP (R interface)
TidyText
Other
Mallet
Google Natural Language API

27. Q & A