NLP_lectures_English

Introduction to Natural Language Processing
in 3 Sessions
Dr. Alexandra M. Liguori
Incubio – The Big Data Academy
Barcelona, March - April, 2015
Dr. Alexandra M. Liguori Introduction to Natural Language Processing in 3 Sessions

Outline: Lecture 1
1 Introduction
2 Natural Language Processing
3 Linguistic Ambiguities
4 Deﬁnition of corpus
5 Typical NLP tasks
6 POS-tagging

Outline: Lecture 2
1 Recap: Typical NLP tasks → practical examples with GATE
2 Def. of semantics
3 Frames approach
1 FrameNet
2 GATE for semantic/content analysis

Outline: Lecture 3
1 Recap: Typical NLP tasks
2 Automatic Question Answering
3 Reference resolution
4 Named Entity Recognition (NER)
5 Keyword / topic / information extraction

Welcome!
Here we go...!!!
Main references:
Text book: Speech and Language Processing by D. Jurafsky
and J. H. Martin
English FrameNet: https://framenet.icsi.berkeley.edu/fndrupal/
Spanish FrameNet: http://sfn.uab.es:8080/SFN
GATE: https://gate.ac.uk/

Introduction: Intelligent machines?
Video:
https://www.youtube.com/watch?v=dSIKBliboIo
(Stanley Kubrick and Arthur C. Clarke,
screenplay of 2001: A Space Odyssey)

Dave Bowman: Open the pod bay doors, HAL.
HAL: I’m sorry Dave, I’m afraid I can’t do that.
(Stanley Kubrick and Arthur C. Clarke,
screenplay of 2001: A Space Odyssey)
https://www.youtube.com/watch?v=dSIKBliboIo

1 Phonetics and phonology

2 Morphology → produce contractions I’m and can’t

3 Syntax → cfr. Open the pod bay doors, HAL.
vs. HAL, the pod bay door is open.
vs. HAL, is the pod bay door open?

4 Lexical semantics → meaning of component words

5 Compositional semantics → knowledge of how
components combine to form larger meanings

6 Pragmatics → cfr. I’m sorry ... , I’m afraid I can’t
vs. No, I won’t open the door.
vs. No.

6 Pragmatics → cfr. I’m sorry ... , I’m afraid I can’t
vs. No, I won’t open the door.
vs. No.
7 Discourse conventions → engaging in structured
conversation using reference that in I’m sorry Dave, I’m
afraid I can’t do that

Natural Language Processing
NLP: techniques that process written human language as
language.

NLP: techniques that process written human language as
language.
Applications
word counting
automatic hyphenation
automated question answering
named entity extraction (NER)
information/content extraction
semantic analysis
sentiment analysis
machine translation

An ideal NLP team is very interdisciplinary, including:
Language experts (linguists)
Maths experts (mathematicians, physicists, statisticians)
Programmers (computer scientists)

NLP: Maths & Computer Science

NLP: Six categories of linguistic knowledge

1 Phonetics and phonology ↔ red - read - read;
sleigh - slay

sleigh - slay
2 Morphology ↔ I/you/we/you/they walk - he/she/it walks;
walked; walking

sleigh - slay
walked; walking
3 Syntax ↔ She ate a mammoth breakfast - She eating a
mammoth breakfast

sleigh - slay
walked; walking
mammoth breakfast
4 Semantics ↔ book (verb) - book (noun);
duck (verb) - duck (noun)

sleigh - slay
walked; walking
mammoth breakfast
4 Semantics ↔ book (verb) - book (noun);
duck (verb) - duck (noun)
5 Pragmatics ↔ open the door - can you open the door? -
could you open the door, please?

6 Discourse

6 Discourse
Gracie: Oh yeah... And then Mr. and Mrs. Jones were
having matrimonial trouble, and my brother was hired to
watch Mrs. Jones.
George: Well, I imagine she was a very attractive woman.
Gracie: She was, and my brother watched her day and
night for six months.
George: Well, what happened?
Gracie: She ﬁnally got a divorce.
George: Mrs. Jones?
Gracie: No, my brother’s wife.

6 Discourse
Gracie: Oh yeah... And then Mr. and Mrs. Jones were
having matrimonial trouble, and my brother was hired to
watch Mrs. Jones.
Gracie: She was, and my brother watched her day and
night for six months.
George: Mrs. Jones?
John went to Bill’s car dealership to check out an
Acura Integra. He looked at it for about an hour.

NLP: Ambiguities and Solutions

Linguistic Ambiguities

Example
I made her duck.

Example
I made her duck.
Five possible interpretations:

Example
I made her duck.
1 I cooked waterfowl for her.

Example
I made her duck.
2 I cooked waterfowl belonging to her.

Example
I made her duck.
3 I created the (plaster?) duck she owns.

Example
I made her duck.
4 I caused her to quickly lower her head or body.

Example
I made her duck.
4 I caused her to quickly lower her head or body.
5 I waved my magic wand and turned her into
undifferentiated waterfowl.

Morphological ambiguity
duck: verb or noun
her: dative pronoun or possessive pronoun

duck: verb or noun
Syntactic ambiguity: make
transitive: taking a single direct object (case 2)
ditransitive: taking two objects, meaning that the ﬁrst object
(her) got made into the second object (duck)
taking a direct object and a verb, meaning that the object
(her) got caused to perform the verbal action (duck)

duck: verb or noun
Syntactic ambiguity: make
transitive: taking a single direct object (case 2)
ditransitive: taking two objects, meaning that the ﬁrst object
(her) got made into the second object (duck)
taking a direct object and a verb, meaning that the object
(her) got caused to perform the verbal action (duck)
Semantic ambiguity: make
cook
create

Corpus
Deﬁnition
Corpus = Large and structured set of texts.

Corpus
Deﬁnition
Corpus = Large and structured set of texts.
NLP
Two types of corpora:
Training corpus ↔ to make the list of rules or to get the
statistical data
Test corpus ↔ to test the results found with the training
corpus

Typical NLP tasks: Basic and simpler tasks

Tokenization

Tokenization RegEx

Tokenization RegEx
Sentence splitting

Tokenization RegEx
Sentence splitting RegEx

Tokenization RegEx
POS-tagging

Tokenization RegEx
POS-tagging
POS-tagging algorithms and
tag sets

Typical NLP tasks: Complex tasks

Lemmatization or Stemming

Implementations of Porter
Stemmer (e.g. in Java),
Stanford NLP tool, GATE, ...

Syntactic parsing

Syntactic parsing
Early algorithm, CYK
algorithm, GHR algorithm,
Stanford Parser (Java
implementation of
probabilistic algorithm)

Syntactic parsing
implementation of
Topic extraction
NER
Semantic analysis
...

Syntactic parsing
implementation of
Topic extraction
NER
Semantic analysis
...
Ad hoc tools, e.g.
dictionaries, ontologies,
Frames, GATE, NLTK,
...

POS-tagging

POS-tagging
Example with Penn Treebank POS-tags:
A/DT woman/NN came/VBD from/IN the/DT back/NN of/IN
the/DT store/NN ./. She/PP appeared/VBD to/TO be/VB
sleepy/JJ and/CC quite/RB a/DT bit/NN younger/JJR than/IN
Mr./NNP Dobbs/NNP and/CC to/TO be/VB wearing/VBG
too/RB much/RB makeup/NN ./.

POS-tagging
Example of ambiguity:

POS-tagging
1 Secretariat/NNP is/VBZ expected/VBN to/TO race/VB
tomorrow/NN ./.

POS-tagging
tomorrow/NN ./.
2 People/NNS continue/VBP to/TO inquire/VB the/DT
reason/NN for/IN the/DT race/NN for/IN outer/JJ space/NN
./.

POS-tagging
Three main tagging algorithms or methods:
1 rule-based tagging, e.g. ENGTWOL
2 stochastic tagging, e.g. HMM tagger
3 transformation-based tagging, e.g. Brill tagger

Rule-based POS-tagging

tomorrow/NN ./.
./.

tomorrow/NN ./.
./.
Large database of hand-written disambiguation rules, e.g.:

tomorrow/NN ./.
./.
Large database of hand-written disambiguation rules, e.g.:
TO + VB → YES
TO + NN → NO
DT + NN → YES
DT + VB → NO

Stochastic POS-tagging

tomorrow/NN ./.
./.

tomorrow/NN ./.
./.
Training corpus to compute probability of given word having
given tag in given context, e.g.:

tomorrow/NN ./.
./.
Training corpus to compute probability of given word having
given tag in given context, e.g.:
is/VBZ expected/VBN to/TO race/VB → 98%
is/VBZ expected/VBN to/TO race/NN → 2%
reason/NN for/IN the/DT race/NN → 97%
reason/NN for/IN the/DT race/VB → 3%

Transformation-based tagging POS-tagging

Secretariat/NNP is/VBZ expected/VBN to/TO race/VB
tomorrow/NN ./.
People/NNS continue/VBP to/TO inquire/VB the/DT

tomorrow/NN ./.
Rules automatically induced from data using Machine Learning
techniques, e.g.:

tomorrow/NN ./.
techniques, e.g.:
1 a priori, prob(race = NN)= 65%, prob(race = VB)= 35%
→ system would always take race = NN

tomorrow/NN ./.
techniques, e.g.:
2 Machine Learning to learn conditional probabilities:

tomorrow/NN ./.
techniques, e.g.:
3 is/VBZ expected/VBN to/TO race/VB → 98%

tomorrow/NN ./.
techniques, e.g.:
3 is/VBZ expected/VBN to/TO race/VB → 98%
4 system takes race = NN or race = VB depending on
context.

POS-tagging
POS-tagging tools for English:
Brill tagger
Stanford Log-linear POS-tagger (Java)
POS-tagger integrated in GATE (Java)
POS-tagger with NLTK (Python)

Topic extraction
NER
Semantic analysis
...

Topic extraction
NER
Semantic analysis
...
Ad hoc tools, e.g.
Frames, GATE, NLTK, ...

EX.: GATE
Concrete examples with GATE:
1 Tokenizer
2 Sentence-splitter
3 POS-tagger
4 Stemmer

EX.: GATE
Concrete examples with GATE:
1 Tokenizer
2 Sentence-splitter
3 POS-tagger
4 Stemmer
GATE
https://gate.ac.uk/
development at the University of Shefﬁeld, UK.

Semantics
’Then you should say what you mean,’
the March Hare went on.
’I do,’ Alice hastily replied;
’at least, I mean what I say –
that’s the same thing, you know.’
’Not the same thing a bit!’ said the Hatter. ’You might just as
well say that
”I see what I eat” is the same thing as ”I eat what I see”! ’
Lewis Carroll,
Alice in Wonderland

Frames and FrameNet
Frame
A schematic representation of a situation involving various
participants, and other conceptual roles.
E.g.:
Abby bought a car from Robin for $5,000.
Robin sold a car to Abby for $5,000.

Frames and FrameNet
Frame
participants, and other conceptual roles.
E.g.:
Abby bought a car from Robin for $5,000.
Robin sold a car to Abby for $5,000.
English FrameNet
https://framenet.icsi.berkeley.edu/fndrupal/
development at the International Computer Science Institute in
Berkeley, California.

English FrameNet Example

Frame Relations
FrameNet additionally captures relationships between different
frames using relations. These include the following:
Inheritance: When one frame is a more specific version of
another, more abstract parent frame. Anything that is true
about the parent frame must also be true about the child
frame, and a mapping is specified between the frame
elements of the parent and the frame elements of the child.
Perspectivized-in: A neutral frame (like
Commerce-transfer-goods) is connected to a frame with a
specific perspective of the same scenario (e.g. the
Commerce-sell frame, which assumes the perspective of
the seller or the Commerce-buy frame, which assumes the
perspective of the buyer)
Subframe: Some frames like the Criminal-process frame
refer to complex scenarios that consist of several individual
states or events that can be described by separate frames
like Arrest, Trial, and so on.

Frame Relations
Precedes: The Precedes relation captures a temporal
order that holds between subframes of a complex scenario.
Causative-of and Inchoative-of: There is a fairly systematic
relationship between stative descriptions (e.g. the
Position-on-a-scale frame, "She had a high salary") and
causative descriptions (Cause-change-of-scalar-position,
"She raised his salary") or inchoative descriptions
(Change-position-on-a-scale, e.g. "Her salary increased").
Using: A relationship that holds between a frame that in
some way involves another frame. For instance, the
Judgment-communication frame uses both the Judgment
frame and the Statement frame, but does not inherit from
either of them because there is no clear correspondence of
the frame elements.
See-also: Connects frames that bear some resemblance
but need to be distinguished carefully.

Spanish FrameNet
Frame
participants, and other conceptual roles. E.g.:
El rock inﬂuye en los artistas de hoy en día
para sus producciones artísticas.
Los artistas de hoy en día se inspiran al rock

Spanish FrameNet
Frame
participants, and other conceptual roles. E.g.:
El rock inﬂuye en los artistas de hoy en día
Los artistas de hoy en día se inspiran al rock
Spanish FrameNet
http://sfn.uab.es:8080/SFN
development at the Universidad Autónoma de Barcelona and
International Computer Science Institute in Berkeley, California.

Spanish FrameNet Example

Frames and GATE
And now...
Ex. in English implementing FRAMES, LUs, and FEs with
GATE !!!

Frames and GATE
And now...
Ex. in English implementing FRAMES, LUs, and FEs with
GATE !!!
GATE
https://gate.ac.uk/
development at the University of Shefﬁeld, UK.

Syntactic parsing
implementation of
Question answering
Topic extraction
NER
Semantic analysis
...

Syntactic parsing
implementation of
Question answering
Topic extraction
NER
Semantic analysis
...
Ad hoc tools, e.g.
Frames, GATE, NLTK, ...

Question Answering
Example from Who’s on ﬁrst

Question Answering

Simple Question Answering
ELIZA
ELIZA simulated a Rogerian psychologist, and could carry on
conversations with the user like the following:
User-1: Men are all alike.
ELIZA-1: IN WHAT WAY
User-2: They’re always bugging us about something or other..
ELIZA-2: CAN YOU THINK OF A SPECIFIC EXAMPLE
User-3: Well, my boyfriend made me come here.
ELIZA-3: YOUR BOYFRIEND MADE YOU COME HERE
User-4: Yes, he says I’m depressed much of the time.
ELIZA-4: I AM SORRY TO HEAR YOU ARE DEPRESSED.

ELIZA
Regular expression substitutions
Eliza worked by having a cascade of regular expression substitutions
that each matched some part of the input lines and changed them:
change all instances of my to YOUR, and I’m to YOU ARE, etc.,
e.g.:

ELIZA
e.g.:
1 User-3: Well, my boyfriend made me come here.

ELIZA
e.g.:
1 User-3: Well, my boyfriend made me come here.
2 User-4: ... I’m depressed ... .
ELIZA-4: ... YOU ARE DEPRESSED.

ELIZA
relevant patterns in the input → creat an appropriate output;
e.g.:

ELIZA
e.g.:
1 s/.* YOU ARE (depressed | sad) .*/I AM SORRY TO HEAR
YOU ARE 1 /

ELIZA
e.g.:
YOU ARE 1 /
2 s/.* YOU ARE (depressed | sad) .*/WHY DO YOU THINK
YOU ARE 1 /

ELIZA
e.g.:
YOU ARE 1 /
YOU ARE 1 /
3 s/.* all .*/IN WHAT WAY/

ELIZA
e.g.:
YOU ARE 1 /
YOU ARE 1 /
3 s/.* all .*/IN WHAT WAY/
4 s/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE/

Reference resolution
Discourse

Discourse
Gracie: Oh yeah... And then Mr. and Mrs. Jones were having
matrimonial trouble, and my brother was hired to watch Mrs. Jones.
Gracie: She was, and my brother watched her day and night for six
months.
George: Mrs. Jones?

Discourse
Gracie: Oh yeah... And then Mr. and Mrs. Jones were having
matrimonial trouble, and my brother was hired to watch Mrs. Jones.
Gracie: She was, and my brother watched her day and night for six
months.
George: Mrs. Jones?
John went to Bill’s car dealership to check out an Acura Integra.
He looked at it for about an hour.

1 Reference phenomena

2 Constraints on coreference

3 Preferences in pronoun interpretation

3 Preferences in pronoun interpretation
4 Example of algorithm for pronoun resolution

Reference phenomena

Reference phenomena
1 Indeﬁnite noun phrases ↔ I saw an Honda Civic today.

Reference phenomena
2 Deﬁnite noun phrases ↔ I saw an Honda Civic today.
The Honda Civic was blue.

Reference phenomena
3 Pronouns ↔ I saw an Honda Civic today. It was blue.

Reference phenomena
4 Demonstratives ↔ I bought an Honda Civic today. It’s
similar to the one I bought ﬁve years ago. That one was
really nice, but I like this one even better.

Reference phenomena
4 Demonstratives ↔ I bought an Honda Civic today. It’s
similar to the one I bought ﬁve years ago. That one was
really nice, but I like this one even better.
5 One-anaphora ↔ I saw no less than 6 Honda Civics
today. Now I want one.

Constraints on coreference

1 number agreement ↔ John has a new car. It is red. /
John has a new car. They are red.

2 person and case agreement ↔ John and Mary have new
cars. They love them.

3 gender agreement ↔ John has a new car. It is attractive.
/ John has a new car. He is attractive.

4 syntactic constraints ↔ John bought himself a new car. /
John bought him a new car.

4 syntactic constraints ↔ John bought himself a new car. /
John bought him a new car.
5 selectional restrictions ↔ John parked his car in the
garage. He had driven it around for hours.

Preferences in pronoun interpretation

1 recency ↔ Peter has an Audi. Bob has a Honda. Anne
likes to drive it.

likes to drive it.
2 grammatical role ↔ Peter went to the Honda dealership
with Bob. He bought a Civic. / Bob went to the Honda
dealership with Peter. He bought a Civic.

likes to drive it.
3 repeated mention ↔ Anne needed a car to drive to her
new job. She decided she wanted something roomy. Carol
went to the Honda dealership with her. She bought a Civic.

likes to drive it.
4 parallelism ↔ Anne went with Carol to the Honda
dealership. Sally went with her to the VW dealership.

likes to drive it.
4 parallelism ↔ Anne went with Carol to the Honda
dealership. Sally went with her to the VW dealership.
5 verb semantics ↔ Peter seized the Honda pamphlet from
Bob. He loves reading about cars. / Peter passed the
Honda pamphlet to Bob. He loves reading about cars.

NER

NER
Named Entity Recognition
Can be broken down in two distinct problems, i.e.:

NER
1 detection of names

NER
2 classiﬁcation of the names by the type of entity to which
they refer → 4 standard types:

NER
1 person (e.g. ”Carol”, ”Tom Hanks”, etc.)

NER
2 organization (e.g. ”WWF”, IBM”, ”Bank of America”, etc.)

NER
3 location (e.g. ”London”, "Washington D.C.”, ”L.A.”,
”Barcelona”, etc.)

NER
3 location (e.g. ”London”, "Washington D.C.”, ”L.A.”,
”Barcelona”, etc.)
4 other (e.g. ”Hotel Sunshine”, etc. )

NER
Tools for Named Entity Recognition
GATE for English, Spanish, and many more, via graphical
interface and Java API (development at the University of
Shefﬁeld, UK)
https://gate.ac.uk/
NETagger: Java based Illinois Named Entity Recognition
(development by Cognitive Computation Group at University of
Illinois at Urbana - Champaign)
http://cogcomp.cs.illinois.edu/page/software_view/NETagger
OpenNLP: rule based and statistical Named Entity Recognition
(development by Apache)
http://opennlp.apache.org/index.html
Stanford CoreNLP: Java-based CRF Named Entity Recognition
(development by Stanford Natural Language Processing Group)
http://nlp.stanford.edu/software/CRF-NER.shtml

Keyword / topic / information extraction
Tools
Keyword extraction: e.g. GATE (ANNIE tool) for English,
Spanish, and many more, via graphical interface and Java
API
→ simply using jape files for the LUs
tool from Volker ?
Topic / information extraction: e.g. GATE (ANNIE tool)
for English, Spanish, and many more, via graphical
interface and Java API
→ using jape files for the LUs, FEs, and FRAMES
GATE
https://gate.ac.uk/
development at the University of Sheffield, UK

Thank you for your attention!
For more information:
Example text book: Speech and Language Processing
by D. Jurafsky and J. H. Martin
Web page: www.alexandramliguoriphd.com
Linkedin proﬁle: Alexandra M. Liguori, Ph.D.

NLP_lectures_English

More Related Content

Viewers also liked

Similar to NLP_lectures_English

More from Alexandra M. Liguori, Ph.D.

NLP_lectures_English