Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fun With Text
Hacking Text Analytics
Cohan Sujay Carlos
Aiaioo Labs
Bangalore
I shall chuck three buzz-words at you:
• One type of big data is unstructured data
• One type of unstructured data is text...
You didn’t answer my question.
Is text analytics important?
Hell, I don’t know!
But what are you doing reading this if it ...
• One Machine Learning Tool
• How to reduce Text Analytics Tasks to
steps which you can solve with merely
this one Machine...
The One ML Tool
Get ready for the one ML tool you’ll need to
hack text analytics
DRUM ROLL !!!!!!!
The One
Get ready for the one ML tool you’ll need to
hack text analytics
The Classifier 
That ML Tool is the Classifier
What is a Classifier?
Something that performs classification.
Classification = categorizing...
Classification = Deciding = Labelling
5’11”
5’ 8”
Classify these door heights as: Short or Tall ?
5’8”
5’11”
6’2”
6’6”
5’ ...
Classification = Deciding = Labelling
5’11” Short
5’ 8” Short
For classification, you always start with
some labelled data...
Classification = Deciding = Labelling
5’11” Short
5’ 8” Short
5’8” Short
5’11” Short
6’2” Tall
6’6” Tall
5’ 2” Short
6’8” ...
Classification = Deciding = Labelling
5’11” Short
5’ 8” Short
In ML, you just provide some examples.
The computer discover...
Classification = Deciding = Labelling
5’11” Short
5’ 8” Short
5’8”
5’11”
6’2” Tall
6’6” Tall
5’ 2” Short
6’8”
6’9”
6’10” T...
Classification = Deciding = Labelling
5’11” Short
5’ 8” Short
5’8” Short
5’11” Short
6’2” Tall
6’6” Tall
5’ 2” Short
6’8” ...
Topic Classification
Can you tell which is about Politics and which is
about Sports?
The United Nations
Security Council t...
Topic Classification
Can you train an ML algorithm to tell which is
about Politics and which is about Sports?
Manchester U...
Topic Classification
Start by taking some samples of documents on
politics and some samples of sports documents.
We’re usi...
Step 1: Learn Multinomial Probabilities
The United States and
Politics
Manchester United
Manchester and Barca
Sports
P(Uni...
One ML Algorithm
Step 1 was easy!!!
Are you ready for Step 2 ?
Step 2: There’s no step 2
P(United/Politics) = 2/7
P(Nations/Politics) = 1/7
P(States/Politics) = 1/7
P(Manchester/Sports)...
One ML Algorithm
Let’s put the classifier to work:
Let’s see if it can classify the following
documents :
1. United Nation...
Running the Topic Classifier
United Nations
P(Politics|United Nations)
= P(United|P)*P(Nations|Politics)*P(Politics)
= (2/...
Running the Topic Classifier
United Nations
P(Politics|United Nations)
>
P(Sports|United Nations)
So, the classifier has r...
Running the Topic Classifier
Manchester United
P(Politics|Manchester United)
= P(Manchester|P)*P(United|P)*P(Politics)
= (...
Running the Topic Classifier
P(Sports|Manchester United)
>
P(Politics|Manchester United)
So, the classifier has returned t...
Topic Classification
Manchester United
Politics
United Nations
Sports
We have successfully used an ML algorithm to
tell us...
Can We Solve Other Problems?
Now, we have an ML tool in our toolkit – a Naïve
Bayesian classifier.
Can We Solve Other Problems?
So, if we can represent a text analysis problem
as a classification task, then we can solve i...
Can We Solve Other Problems?
So, if we can represent a text analysis problem
as a classification task, then we can solve i...
Problem 1: Sentence Segmentation
The problem of identifying the end-points of
sentences is called sentence segmentation. I...
Sentence Segmentation
Yes, Mr. Anurag. You need a D.L. to drive.
Sentence segmentation
Classify each ‘.’, ‘!’ and ‘?’ into...
Sentence Segmentation
Hurray! We have turned the problem
of sentence segmentation into a
classification problem.
Once you ...
Sentence Segmentation
FIND THE FEATURES!
Features
Which country’s flag is this?
Features are the clues you need for
good decision making.
Features
Which country’s flag is this?
One important feature for solving this
decision problem is colour.
Features for Topic Classification
P(United/Politics) = 2/7
P(Nations/Politics) = 1/7
P(States/Politics) = 1/7
P(Manchester...
Sentence Segmentation
What are the features you would use?
Yes, Mr. Anurag. You need a D.L. to drive.
That’s when you star...
Sentence Segmentation
What are the features you would use?
Is_This_Character_a_Dot
Is_This_Within_Quotes
Is_Next_Letter_Ca...
Sentence Segmentation
Train an NB classifier on some text
whose characters are marked as
sentence terminators or not.
It w...
Problem 2: Tokenization
Now, get a D.L., Mr. Anurag.
For almost all analyses, you have to identify the
individual words in...
Tokenization
Yes, Mr. Anurag. You need a D.L. to drive.
Tokenization
Classify each ‘.’, ‘!’, ‘,’, ‘ ’, ‘?’ etc. into:
1) w...
Tokenization or Word Segmentation
We have turned the problem of word
segmentation into a classification
problem.
Train a c...
Problem 3: Part of Speech (POS) Tagging
Anurag needs a D.L.
Anurag/NNP needs/VBZ a/DT D.L./NN
How do you tag the words in ...
Problem 3: Part of Speech (POS) Tagging
Anurag needs a D.L.
Anurag/NNP needs/VBZ a/DT D.L./NN
POS Tagging
Run from the fir...
We have turned the problem of POS Tagging into a
Classification problem where you label each word as:
1. Noun
2. Verb
3. A...
Problem 4: Named Entity Recognition
Anurag is looking for a Hyundai car in Bangalore
Anurag = Person
Hyundai car = Vehicle...
Problem 4: Named Entity Recognition
Anurag is looking for a Hyundai car in Bangalore
Anurag = Person
Hyundai car = Vehicle...
We have turned the problem of Named Entity
Recognition (NER) into a Tagging problem
where you label each word as:
1. Perso...
But we know that we can turn the problem of
Tagging into a Classification problem over the
same labels:
1. Person
2. Vehic...
Problem 4: Named Entity Recognition
Anurag is looking for a Hyundai car in Bangalore
Anurag/Person is/Other looking/Other
...
Hey We Have One More ML Tool
We just built one more very useful ML tool!
The Extractor 
Extraction
What is an Extractor?
Something that performs extraction.
Extraction = recognizing
Extraction = finding
Extract...
Problem 5: Relation Extraction
Tim Cook is the new CEO of Apple Computers
Relation: CEO_of
Tim Cook (Person) Apple Compute...
Extracting Meaning
Text: Tim Cook is the new CEO of Apple Computers
Analysis: Tim/Person Cook/Person is the new CEO
of App...
Extracting Meaning
Text: Tim Cook is the new CEO of Apple Computers
Analysis: Tim/Person Cook/Person is the new CEO
of App...
Extracting Meaning
Text: Tim Cook is the new CEO of Apple Computers
Analysis: Tim/Person Cook/Person is the new CEO
of App...
But look at what we’re extracting!
Do you realize what we’re extracting?
Meaning !!! 
Extracting Meaning
Text: I am looking for a Hyundai car in B’lore
Syntactic Analysis: I/Pronoun am/BE looking/V
for a/DT H...
But is this comprehensive?
Are you telling me that this is all I have to do to
extract every possible sort of simple meani...
Intentional: I want to buy a computer.
Information: There was heavy snowfall in Sikkim.
Two Kinds of Utterances
360 Degree...
I want to buy a computer.
How do you deal
with intentional
utterances
360 Degree Text
Analysis
Intentional Utterance
Raw Text: Are you sad that Steve Jobs died?
Analysis: This person is inquiring about
someone’s emoti...
Subjective Utterance
Raw Text: I am sad that Steve Jobs died
Analysis: This person holds a positive opinion
on Steve Jobs
...
How do you deal
with informational
utterances
360 Degree Text
Analysis
There was heavy snowfall in Sikkim.
Approaches to
Extracting Meaning
Raw Text: There is heavy snowfall in Sikkim.
Analysis: Snowfall event
Event Analysis
Even...
Approaches to
Extracting Meaning
Raw Text: Bangalore is the capital of K’taka
Analysis: capital_of relation exists
Fact An...
Extracting Meaning
Text: I am looking for a Hyundai car in Bangalore
Semantic Analysis: I/Person am looking for a
Hyundai/...
Extracting Meaning
Text: Tim Cook is the new CEO of Apple Computers
Analysis: Tim/Person Cook/Person is the new CEO
of App...
So, I can do anything (badly?)
I got it! I got it! I can do anything.
But how do I know how well I am doing it ? 
Measurement
Measurement:
This measurement thing is very
important in any design process,
because …
Measurement
How do you measure the performance of a
classifier?
Measurement:
… it lets you compare two designs and
decide ...
Before we go on …
… promise me …
… that you will never forget what I am about to
tell you … 
Measurement
Break up the data points into training
and test parts (usually an 80:20 split)
and never test on your training...
Measurement
Why not test on the training data?
Break up the data points into training
and test – usually an 80:20 split.
T...
Measurement
Why not develop on the test data?
Or if you are still developing features,
break up the data points into train...
That applies to college as well …
… you won’t get accurate measurements …
… if you test students using questions that
appe...
Classification Quality Metric - Accuracy
Correct Answers
Total Number of Questions
Classification Quality Metric - Accuracy
Politics Documents classified as Politics
+ Sports Documents classified as Sports...
Classification Quality Metric - Accuracy
Gold - Politics Gold – Sports
Observed - Politics TN (True Negative) FN (False Po...
Accuracy
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TN = 990 FN = 100
Observed – Politics FP = 10 TP = ...
Accuracy
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TN = 990 FN = 100
Observed – Politics FP = 10 TP = ...
Classification Quality Metric - Recall
How many Politics document did it find
Total number of Politics documents
in the te...
Recall
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TN = 990 FN = 100
Observed – Politics FP = 10 TP = 90...
Recall
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TN = 990 FN = 100
Observed – Politics FP = 10 TP = 90...
Recall
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TP = 990 FP = 100
Observed – Politics FN = 10 TN = 90...
Recall
Gold – Sports (1000) Gold – Politics (1000)
Observed – Sports TP = 990 FP = 100
Observed – Politics FN = 10 TN = 90...
Classification Quality Metric - Precision
How many were really Politics documents
Total number of documents the
classifier...
Precision
Point of View - Politics
= ?
Gold – Sports (1040) Gold – Politics (960)
Observed – Sports TN = 990 FN = 10
Obser...
Precision
Point of View - Politics
= 95%
Gold – Sports (1040) Gold – Politics (960)
Observed – Sports TN = 990 FN = 10
Obs...
Precision
Point of View - Sports
= ?
Gold – Sports (1040) Gold – Politics (960)
Observed – Sports TP = 990 FP = 10
Observe...
Precision
Point of View - Sports
= 99%
Gold – Sports (1040) Gold – Politics (960)
Observed – Sports TP = 990 FP = 10
Obser...
Are we done?
… kinda …
The more training you have, the better you will
get at text analysis … so keep learning!
How to Learn More?
Grab the UC Berkeley Natural Language
Processing Course’s slides. The course’s
name is CS 294.
Start re...
Now, are we done?
… kinda …
Have Fun!
… and I have an exercise for you …
Did you know
… that you can use a classifier …
… to do clustering?
The exercise is … think about it …
THE END
Fun With Text
Hacking Text Analytics
Cohan Sujay Carlos
Aiaioo Labs
Bangalore
The bottom
Upcoming SlideShare
Loading in …5
×

Fun with Text - Hacking Text Analytics

2,343 views

Published on

"Fun with Text - Hacking Text Analytics" is a tutorial for beginners to text analytics. It explains how certain text analytics problems can be reduced to one machine learning problem (classification). For instance, it tells you how segmentation tasks can be reduced to classification tasks. The tutorial also explains how a naive bayesian classifier can be crafted in case you don't have one.

Published in: Data & Analytics

Fun with Text - Hacking Text Analytics

  1. 1. Fun With Text Hacking Text Analytics Cohan Sujay Carlos Aiaioo Labs Bangalore
  2. 2. I shall chuck three buzz-words at you: • One type of big data is unstructured data • One type of unstructured data is text • The analysis of text is text analysis Is text analytics important?
  3. 3. You didn’t answer my question. Is text analytics important? Hell, I don’t know! But what are you doing reading this if it isn’t?
  4. 4. • One Machine Learning Tool • How to reduce Text Analytics Tasks to steps which you can solve with merely this one Machine Learning Tool What we’re going to cover
  5. 5. The One ML Tool Get ready for the one ML tool you’ll need to hack text analytics DRUM ROLL !!!!!!!
  6. 6. The One Get ready for the one ML tool you’ll need to hack text analytics The Classifier 
  7. 7. That ML Tool is the Classifier What is a Classifier? Something that performs classification. Classification = categorizing Classification = deciding Classification = labelling Classification = Deciding = Labelling
  8. 8. Classification = Deciding = Labelling 5’11” 5’ 8” Classify these door heights as: Short or Tall ? 5’8” 5’11” 6’2” 6’6” 5’ 2” 6’8” 6’9” 6’10”
  9. 9. Classification = Deciding = Labelling 5’11” Short 5’ 8” Short For classification, you always start with some labelled data points. 5’8” 5’11” 6’2” Tall 6’6” Tall 5’ 2” Short 6’8” 6’9” 6’10” Tall
  10. 10. Classification = Deciding = Labelling 5’11” Short 5’ 8” Short 5’8” Short 5’11” Short 6’2” Tall 6’6” Tall 5’ 2” Short 6’8” Tall 6’9” Tall 6’10” Tall If you were doing analysis, you’d ask a human to come up with a rule like: A door below 6’ is Short else it’s Tall
  11. 11. Classification = Deciding = Labelling 5’11” Short 5’ 8” Short In ML, you just provide some examples. The computer discovers the rule. 5’8” 5’11” 6’2” Tall 6’6” Tall 5’ 2” Short 6’8” 6’9” 6’10” Tall
  12. 12. Classification = Deciding = Labelling 5’11” Short 5’ 8” Short 5’8” 5’11” 6’2” Tall 6’6” Tall 5’ 2” Short 6’8” 6’9” 6’10” Tall The ML algorithm learns something like: A door below 6’ is Short else it’s Tall
  13. 13. Classification = Deciding = Labelling 5’11” Short 5’ 8” Short 5’8” Short 5’11” Short 6’2” Tall 6’6” Tall 5’ 2” Short 6’8” Tall 6’9” Tall 6’10” Tall You will learn to create an ML algorithm that learns something like this and that works with text.
  14. 14. Topic Classification Can you tell which is about Politics and which is about Sports? The United Nations Security Council today Manchester United beat Barca to reach
  15. 15. Topic Classification Can you train an ML algorithm to tell which is about Politics and which is about Sports? Manchester United beat Barca to reach Politics Sports The United Nations Security Council today
  16. 16. Topic Classification Start by taking some samples of documents on politics and some samples of sports documents. We’re using really short documents so you can do all the calculations manually & see that this ML algorithm really works! Politics Sports The United Nations The United States and Manchester United Manchester and Barca
  17. 17. Step 1: Learn Multinomial Probabilities The United States and Politics Manchester United Manchester and Barca Sports P(United/Politics) = 2/7 The United Nations P(Nations/Politics) = 1/7 P(States/Politics) = 1/7 P(Manchester/Sports) = 2/5 P(United/Sports) = 1/5 P(Barca/Sports) = 1/5 P(Politics) = 7/12 P(Sports) = 5/12 P(The/Politics) = 2/7 P(and/Sports) = 1/5 P(and/Politics) = 1/7
  18. 18. One ML Algorithm Step 1 was easy!!! Are you ready for Step 2 ?
  19. 19. Step 2: There’s no step 2 P(United/Politics) = 2/7 P(Nations/Politics) = 1/7 P(States/Politics) = 1/7 P(Manchester/Sports) = 2/5 P(United/Sports) = 1/5 P(Barca/Sports) = 1/5 P(Politics) = 7/12 P(Sports) = 5/12 P(The/Politics) = 2/7 P(and/Sports) = 1/5 P(and/Politics) = 1/7 This is a Naïve Bayesian Classifier!!!
  20. 20. One ML Algorithm Let’s put the classifier to work: Let’s see if it can classify the following documents : 1. United Nations 2. Manchester United We are using deliberately short documents!
  21. 21. Running the Topic Classifier United Nations P(Politics|United Nations) = P(United|P)*P(Nations|Politics)*P(Politics) = (2/7)*(1/7)*(7/12) = 1/(7*6) P(Sports|United Nations) = P(United|S)*P(Nations|Sports)*P(Sports) = (1/5)*(0)*(5/12) = 0
  22. 22. Running the Topic Classifier United Nations P(Politics|United Nations) > P(Sports|United Nations) So, the classifier has returned the category POLITICS
  23. 23. Running the Topic Classifier Manchester United P(Politics|Manchester United) = P(Manchester|P)*P(United|P)*P(Politics) = (0)*(2/7)*(7/12) = 0 P(Sports|Manchester United) = P(Manchester|S)*P(United|S)*P(Sports) = (2/5)*(1/5)*(5/12) = 2/(5*12)
  24. 24. Running the Topic Classifier P(Sports|Manchester United) > P(Politics|Manchester United) So, the classifier has returned the category SPORTS Manchester United
  25. 25. Topic Classification Manchester United Politics United Nations Sports We have successfully used an ML algorithm to tell us which document is about Politics and which is about Sports !!!
  26. 26. Can We Solve Other Problems? Now, we have an ML tool in our toolkit – a Naïve Bayesian classifier.
  27. 27. Can We Solve Other Problems? So, if we can represent a text analysis problem as a classification task, then we can solve it using ML. Now, we have an ML tool in our toolkit – a Naïve Bayesian classifier.
  28. 28. Can We Solve Other Problems? So, if we can represent a text analysis problem as a classification task, then we can solve it using ML. Now, we have an ML tool in our toolkit – a Naïve Bayesian classifier. So, let us learn how to represent text analysis problems as classification tasks.
  29. 29. Problem 1: Sentence Segmentation The problem of identifying the end-points of sentences is called sentence segmentation. It can be reduced to a classification problem.
  30. 30. Sentence Segmentation Yes, Mr. Anurag. You need a D.L. to drive. Sentence segmentation Classify each ‘.’, ‘!’ and ‘?’ into: 1) sentence terminator and 2) not a sentence terminator.
  31. 31. Sentence Segmentation Hurray! We have turned the problem of sentence segmentation into a classification problem. Once you have modeled a text analytics problem as an ML problem, there is one more step you need to perform to get a working solution.
  32. 32. Sentence Segmentation FIND THE FEATURES!
  33. 33. Features Which country’s flag is this? Features are the clues you need for good decision making.
  34. 34. Features Which country’s flag is this? One important feature for solving this decision problem is colour.
  35. 35. Features for Topic Classification P(United/Politics) = 2/7 P(Nations/Politics) = 1/7 P(States/Politics) = 1/7 P(Manchester/Sports) = 2/5 P(United/Sports) = 1/5 P(Barca/Sports) = 1/5 P(Politics) = 7/12 P(Sports) = 5/12 P(The/Politics) = 2/7 P(and/Sports) = 1/5 P(and/Politics) = 1/7 For topic classification, the features are the individual words in the text.
  36. 36. Sentence Segmentation What are the features you would use? Yes, Mr. Anurag. You need a D.L. to drive. That’s when you start reading the research papers!!!
  37. 37. Sentence Segmentation What are the features you would use? Is_This_Character_a_Dot Is_This_Within_Quotes Is_Next_Letter_Capitalized Is_Prev_Letter_Capitalized Number_of_Words_in_Sentence_so_Far Is_Next_Word_a_Name Yes, Mr. Anurag. You need a D.L. to drive.
  38. 38. Sentence Segmentation Train an NB classifier on some text whose characters are marked as sentence terminators or not. It will learn to assign characters to the categories word terminator and not a word terminator.
  39. 39. Problem 2: Tokenization Now, get a D.L., Mr. Anurag. For almost all analyses, you have to identify the individual words in the text. How can you break a sentence into words? Now , get a D.L. , Mr. Anurag .
  40. 40. Tokenization Yes, Mr. Anurag. You need a D.L. to drive. Tokenization Classify each ‘.’, ‘!’, ‘,’, ‘ ’, ‘?’ etc. into: 1) word terminator and 2) not a word terminator.
  41. 41. Tokenization or Word Segmentation We have turned the problem of word segmentation into a classification problem. Train a classifier on text whose characters are marked as word terminators or not. It will learn to assign characters to the categories word terminator and not a word terminator.
  42. 42. Problem 3: Part of Speech (POS) Tagging Anurag needs a D.L. Anurag/NNP needs/VBZ a/DT D.L./NN How do you tag the words in a sentence?
  43. 43. Problem 3: Part of Speech (POS) Tagging Anurag needs a D.L. Anurag/NNP needs/VBZ a/DT D.L./NN POS Tagging Run from the first to the last word in the sentence, classifying each word in the sentence into a part-of-speech category.
  44. 44. We have turned the problem of POS Tagging into a Classification problem where you label each word as: 1. Noun 2. Verb 3. Adjective 4. Adverb 5. Interjection 6. Conjunction (and, or, if, neither … nor) 7. Pronoun (I, it, you, them) 8. Preposition (in, of, out) Problem 3: Part of Speech (POS) Tagging
  45. 45. Problem 4: Named Entity Recognition Anurag is looking for a Hyundai car in Bangalore Anurag = Person Hyundai car = Vehicle Bangalore = Place How do you extract the named entities in a sentence using a classifier?
  46. 46. Problem 4: Named Entity Recognition Anurag is looking for a Hyundai car in Bangalore Anurag = Person Hyundai car = Vehicle Bangalore = Place Anurag/Person is/Other looking/Other for/Other a/Other Hyundai/Vehicle car/Vehicle in/Other Bangalore/Place
  47. 47. We have turned the problem of Named Entity Recognition (NER) into a Tagging problem where you label each word as: 1. Person 2. Vehicle 3. Place Problem 4: Named Entity Recognition
  48. 48. But we know that we can turn the problem of Tagging into a Classification problem over the same labels: 1. Person 2. Vehicle 3. Place Problem 4: Named Entity Recognition
  49. 49. Problem 4: Named Entity Recognition Anurag is looking for a Hyundai car in Bangalore Anurag/Person is/Other looking/Other for/Other a/Other Hyundai/Vehicle car/Vehicle in/Other Bangalore/Place Named Entity Recognition Run from the first to the last word in the sentence, classifying each word in the sentence into a Named Entity Recognition category.
  50. 50. Hey We Have One More ML Tool We just built one more very useful ML tool! The Extractor 
  51. 51. Extraction What is an Extractor? Something that performs extraction. Extraction = recognizing Extraction = finding Extraction = locating Extraction = Finding = Locating
  52. 52. Problem 5: Relation Extraction Tim Cook is the new CEO of Apple Computers Relation: CEO_of Tim Cook (Person) Apple Computers (Org) How do you identify relations between entities in a sentence using a classifier?
  53. 53. Extracting Meaning Text: Tim Cook is the new CEO of Apple Computers Analysis: Tim/Person Cook/Person is the new CEO of Apple/Org Computers/Org Relation Extraction Step 1:
  54. 54. Extracting Meaning Text: Tim Cook is the new CEO of Apple Computers Analysis: Tim/Person Cook/Person is the new CEO of Apple/Org Computers/Org Relation Extraction Step 1: Step 2: Relation Extraction {Tim/Person Cook/Person, Apple/Org Computers/Org} => CEO_of
  55. 55. Extracting Meaning Text: Tim Cook is the new CEO of Apple Computers Analysis: Tim/Person Cook/Person is the new CEO of Apple/Org Computers/Org Relation Extraction Step 1: Step 2: Relation Extraction Run through all pairs of named entities, classifying each pair into CEO_of or Other.
  56. 56. But look at what we’re extracting! Do you realize what we’re extracting? Meaning !!! 
  57. 57. Extracting Meaning Text: I am looking for a Hyundai car in B’lore Syntactic Analysis: I/Pronoun am/BE looking/V for a/DT Hyundai/NP car/NN in/PP Bangalore/NP Semantic Analysis: I/Person am looking for a Hyundai/Vehicle car/Vehicle in Bangalore/Place
  58. 58. But is this comprehensive? Are you telling me that this is all I have to do to extract every possible sort of simple meaning? Yeah!!!  All utterances fall into two main categories …
  59. 59. Intentional: I want to buy a computer. Information: There was heavy snowfall in Sikkim. Two Kinds of Utterances 360 Degree Text Analysis
  60. 60. I want to buy a computer. How do you deal with intentional utterances 360 Degree Text Analysis
  61. 61. Intentional Utterance Raw Text: Are you sad that Steve Jobs died? Analysis: This person is inquiring about someone’s emotions concerning Steve Jobs Intention Analysis Intention Holder: I Intention: inquire
  62. 62. Subjective Utterance Raw Text: I am sad that Steve Jobs died Analysis: This person holds a positive opinion on Steve Jobs Sentiment Analysis Sentiment Holder: I Object of Sentiment: Steve Jobs Polarity of Sentiment: positive
  63. 63. How do you deal with informational utterances 360 Degree Text Analysis There was heavy snowfall in Sikkim.
  64. 64. Approaches to Extracting Meaning Raw Text: There is heavy snowfall in Sikkim. Analysis: Snowfall event Event Analysis Event: snowfall
  65. 65. Approaches to Extracting Meaning Raw Text: Bangalore is the capital of K’taka Analysis: capital_of relation exists Fact Analysis Entity: Bangalore/Place Karnataka/Place Relation: Bangalore capital_of K’taka
  66. 66. Extracting Meaning Text: I am looking for a Hyundai car in Bangalore Semantic Analysis: I/Person am looking for a Hyundai/Thing car/Thing in Bangalore/Place Entity Extraction Entities: I Person Hyundai car Thing Bangalore Place Fact Analysis
  67. 67. Extracting Meaning Text: Tim Cook is the new CEO of Apple Computers Analysis: Tim/Person Cook/Person is the new CEO of Apple/Org Computers/Org Relation Extraction Relation: CEO_of Tim Cook (Person) Apple Computers(Org) Fact Analysis
  68. 68. So, I can do anything (badly?) I got it! I got it! I can do anything. But how do I know how well I am doing it ? 
  69. 69. Measurement Measurement: This measurement thing is very important in any design process, because …
  70. 70. Measurement How do you measure the performance of a classifier? Measurement: … it lets you compare two designs and decide which is better.
  71. 71. Before we go on … … promise me … … that you will never forget what I am about to tell you … 
  72. 72. Measurement Break up the data points into training and test parts (usually an 80:20 split) and never test on your training data.
  73. 73. Measurement Why not test on the training data? Break up the data points into training and test – usually an 80:20 split. Train on 80% Test on the remaining 20%
  74. 74. Measurement Why not develop on the test data? Or if you are still developing features, break up the data points into training, development and test – usually a 70:10:20 split. Train on 70% Develop on 10% Test on the remaining 20%
  75. 75. That applies to college as well … … you won’t get accurate measurements … … if you test students using questions that appeared in the question bank! … and you will be encouraging students to learn things by rote!
  76. 76. Classification Quality Metric - Accuracy Correct Answers Total Number of Questions
  77. 77. Classification Quality Metric - Accuracy Politics Documents classified as Politics + Sports Documents classified as Sports Total Number of Documents If your categories are Politics and Sports
  78. 78. Classification Quality Metric - Accuracy Gold - Politics Gold – Sports Observed - Politics TN (True Negative) FN (False Positive) Observed - Sports FP (False Negative) TP (True Positive) Point of View = Sports = (+ve)
  79. 79. Accuracy Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TN = 990 FN = 100 Observed – Politics FP = 10 TP = 900 Point of View - Politics = ?
  80. 80. Accuracy Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TN = 990 FN = 100 Observed – Politics FP = 10 TP = 900 Point of View - Politics = 94.5%
  81. 81. Classification Quality Metric - Recall How many Politics document did it find Total number of Politics documents in the test data
  82. 82. Recall Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TN = 990 FN = 100 Observed – Politics FP = 10 TP = 900 Point of View - Politics = ?
  83. 83. Recall Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TN = 990 FN = 100 Observed – Politics FP = 10 TP = 900 Point of View - Politics = 90%
  84. 84. Recall Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TP = 990 FP = 100 Observed – Politics FN = 10 TN = 900 Point of View - Sports = ?
  85. 85. Recall Gold – Sports (1000) Gold – Politics (1000) Observed – Sports TP = 990 FP = 100 Observed – Politics FN = 10 TN = 900 Point of View - Sports = 99%
  86. 86. Classification Quality Metric - Precision How many were really Politics documents Total number of documents the classifier identified as Politics
  87. 87. Precision Point of View - Politics = ? Gold – Sports (1040) Gold – Politics (960) Observed – Sports TN = 990 FN = 10 Observed – Politics FP = 50 TP = 950
  88. 88. Precision Point of View - Politics = 95% Gold – Sports (1040) Gold – Politics (960) Observed – Sports TN = 990 FN = 10 Observed – Politics FP = 50 TP = 950
  89. 89. Precision Point of View - Sports = ? Gold – Sports (1040) Gold – Politics (960) Observed – Sports TP = 990 FP = 10 Observed – Politics FN = 50 TN = 950
  90. 90. Precision Point of View - Sports = 99% Gold – Sports (1040) Gold – Politics (960) Observed – Sports TP = 990 FP = 10 Observed – Politics FN = 50 TN = 950
  91. 91. Are we done? … kinda … The more training you have, the better you will get at text analysis … so keep learning!
  92. 92. How to Learn More? Grab the UC Berkeley Natural Language Processing Course’s slides. The course’s name is CS 294. Start reading the ACL conference’s research papers (ACL = Association of Computational Linguistics)
  93. 93. Now, are we done? … kinda … Have Fun! … and I have an exercise for you …
  94. 94. Did you know … that you can use a classifier … … to do clustering? The exercise is … think about it …
  95. 95. THE END Fun With Text Hacking Text Analytics Cohan Sujay Carlos Aiaioo Labs Bangalore The bottom

×