Polling the Blogosphere: a Rule-Based Approach to Belief Classification Jason Kessler Indiana University, Bloomington

Belief Analysis of Blogs Polling the blogosphere on a controversial proposition Literal search on a proposition (e.g., “Obama is electable”) Which blog entries contain assert it? Which deny it? Aggregate results 243 bloggers assert it 616 bloggers deny it

Polling the blogosphere on a controversial proposition

Literal search on a proposition (e.g., “Obama is electable”)

Which blog entries contain assert it? Which deny it?

Aggregate results

243 bloggers assert it

616 bloggers deny it

Motivating Example Polling for “the Moon landings were staged” “ The theory that the Moon landings were staged is complete nonsense.” The writer denies “the Moon landings were staged.”

Polling for “the Moon landings were staged”

“ The theory that the Moon landings were staged is complete nonsense.”

The writer denies “the Moon landings were staged.”

Motivating Example If Obama is electable , the country is in good shape. Writer takes no stance toward “Obama is electable”.

If Obama is electable , the country is in good shape.

Writer takes no stance toward “Obama is electable”.

Problem When a writer uses a declarative finite clause, does that writer assert, deny, or take no stance toward its truth value? This is the problem of identifying a writer’s stance toward a proposition . Veridicity or facticity of a proposition.

When a writer uses a declarative finite clause, does that writer assert, deny, or take no stance toward its truth value?

This is the problem of identifying a writer’s stance toward a proposition .

Veridicity or facticity of a proposition.

Example Everybody is sad that the bar closed. The writer asserts “the bar closed.” Belief != Sentiment Negative sentiment toward “the bar closed” Positive stance.

Everybody is sad that the bar closed.

The writer asserts “the bar closed.”

Belief != Sentiment

Negative sentiment toward “the bar closed”

Positive stance.

Outline System Description Given a proposition, sentence Dependency Parse Syntactic Representation Hand written patterns over semantic classes Veridicality Elements Veridicality Transformations Evaluation Proof of concept Promising results

System Description

Given a proposition, sentence

Dependency Parse

Syntactic Representation

Hand written patterns over semantic classes

Veridicality Elements

Veridicality Transformations

Evaluation

Proof of concept

Promising results

Dependency Parse Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations The theory that the Moon landings were staged is complete nonsense.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Veridicality Elements (VEs) Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations The theory that the Moon landings were staged is complete nonsense.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Veridicality Transformations (VTs) Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations The theory that the Moon landings were staged is complete nonsense.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Veridicality Transformations (VTs) Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations The theory that the Moon landings were staged is complete nonsense.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

System Structure: Veridicality Elements Find expressions that have the potential of changing the truth-value of a proposition or referring to it Different classes affect truth values differently Examples: Assertion – Positive The assertion that the sky is blue Nonsense – Negative The idea that the sky is orange is nonsense If – Neutral Pretend – Counter-factive

Find expressions that have the potential of changing the truth-value of a proposition or referring to it

Different classes affect truth values differently

Examples:

Assertion – Positive

The assertion that the sky is blue

Nonsense – Negative

The idea that the sky is orange is nonsense

If – Neutral

Pretend – Counter-factive

Finding Veridicality Elements Manually created seed sets Search web for patterns likely to contain VEs “ I agree with the assertion that” “ I * with the assertion that” “ I quibble with the assertion that” “ I take issue with the assertion that” Manually classify matches, form new queries “ I take issue with the * that” “ I take issue with the argument that” Similar to Brin (1998)

Manually created seed sets

Search web for patterns likely to contain VEs

“ I agree with the assertion that”

“ I * with the assertion that”

“ I quibble with the assertion that”

“ I take issue with the assertion that”

Manually classify matches, form new queries

“ I take issue with the * that”

“ I take issue with the argument that”

Similar to Brin (1998)

System Structure: Veridicality Transformations Relate these expressions to propositions Some expressions won’t be related to propositions Why bag-of-Veridicality-Elements fails Templates over dependency graphs Select for a VE class and a proposition

Relate these expressions to propositions

Some expressions won’t be related to propositions

Why bag-of-Veridicality-Elements fails

Templates over dependency graphs

Select for a VE class and a proposition

System Structure: Veridicality Transformations Examples Expression is a main verb, proposition is its comp. clause John pretended the monkey was harmless . Cleft construction, expression is an adjective It is inconceivable that two plus two equals five .

Examples

Expression is a main verb, proposition is its comp. clause

John pretended the monkey was harmless .

Cleft construction, expression is an adjective

It is inconceivable that two plus two equals five .

Another Example Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations If Bob goes to school, he realizes the Earth is round.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Another Example Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations If Bob goes to school, he realizes the Earth is round.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Another Example Pipeline Stages: Dependency Parse Tag Veridicality Elements Apply Veridicality Transformations If Bob goes to school, he realizes the Earth is round.

Pipeline Stages:

Dependency Parse

Tag Veridicality Elements

Apply Veridicality Transformations

Evaluation Primitive, proof-of-concept evaluation Can we poll the blogosphere ? Google blog search for “abortion is murder” Unseen data Run the system on the first 100 hits. See if it does better baseline.

Primitive, proof-of-concept evaluation

Can we poll the blogosphere ?

Google blog search for “abortion is murder”

Unseen data

Run the system on the first 100 hits.

See if it does better baseline.

Evaluation Exclude a number of results: Spam blogs Long, unparsable sentences Trivial sentences (no VEs) Abortion is murder! Questions

Exclude a number of results:

Spam blogs

Long, unparsable sentences

Trivial sentences (no VEs)

Abortion is murder!

Questions

Evaluation Corpus Statistics: 48 Sentences 27 positive 3 negative 18 neutral 39 classified correctly (81% accuracy) Majority class was positive, giving a baseline of 56% accuracy

Corpus Statistics:

48 Sentences

27 positive

3 negative

18 neutral

39 classified correctly (81% accuracy)

Majority class was positive, giving a baseline of 56% accuracy

Related Work Nairn et al. (2006) focused on main verbs Complex behavior under negation Work on contextual polarity for sentiment analysis. Wilson et al. (2005) Statistical approach Polanyi and Zaenen (2006) Theoretical approach

Nairn et al. (2006) focused on main verbs

Complex behavior under negation

Work on contextual polarity for sentiment analysis.

Wilson et al. (2005)

Statistical approach

Polanyi and Zaenen (2006)

Theoretical approach

Related Work Somasundaran et al. (2007) Statistical techniques used to detect presence of “arguing” in a sentence. Arguing = writer takes a non-neutral stance toward some content

Somasundaran et al. (2007)

Statistical techniques used to detect presence of “arguing” in a sentence.

Arguing = writer takes a non-neutral stance toward some content

Future Work Annotate corpus Further testing Statistical approaches Augment VE/VTs Integrate Nairn et al. (2006) Take into account questions

Annotate corpus

Further testing

Statistical approaches

Augment VE/VTs

Integrate Nairn et al. (2006)

Take into account questions

Takeaways Belief analysis is a young field Bag-of-words is not enough Shallow linguistic methods show promise

Belief analysis is a young field

Bag-of-words is not enough

Shallow linguistic methods show promise

Questions? Thank you. References: Brin, S. 1998. Extracting patterns and relations from the world wide web. In WebDB Workshop at 6 th International Conference on Extending Database Technology, EDBT’98. Nairn, R.; Condoravdi, C.; and Karttunen, L. 2006. Computing relative polarity for textual inference. In ICoS-5 . Polanyi, L.; and Zaenen, A. 2005. Contextual valence shifters. In Shanahan, J. G.; Qu, Y.; and Wiebe J., eds,. Computing Attitude and Affect in Text. Somasundaran, S.; Wilson, T.; Wiebe, J.; and Stoyanov, V. 2007. QA with attitude: Exploiting opinion type analysis for improving question answering in on-line discussions and the news. In ICWSM. Wilson, T.; Wiebe, J.; and Hoffmann, P. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP.

Thank you.

References:

Brin, S. 1998. Extracting patterns and relations from the world wide web. In WebDB Workshop at 6 th International Conference on Extending Database Technology, EDBT’98.

Nairn, R.; Condoravdi, C.; and Karttunen, L. 2006. Computing relative polarity for textual inference. In ICoS-5 .

Polanyi, L.; and Zaenen, A. 2005. Contextual valence shifters. In Shanahan, J. G.; Qu, Y.; and Wiebe J., eds,. Computing Attitude and Affect in Text.

Somasundaran, S.; Wilson, T.; Wiebe, J.; and Stoyanov, V. 2007. QA with attitude: Exploiting opinion type analysis for improving question answering in on-line discussions and the news. In ICWSM.

Wilson, T.; Wiebe, J.; and Hoffmann, P. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP.

Implementation Veridicality Element Classes:

Veridicality Transformations