The document proposes a method to estimate the intensity of contradiction between reviews related in time by considering both the sentiment polarity and ratings associated with identified aspects in the reviews. It clusters reviews by session, identifies aspects, detects sentiment polarity, and measures contradiction intensity by calculating the dispersion of review-aspect points from their centroid. An evaluation on real online course reviews shows correlations between the method's results and human judgments, with higher correlation when the sentiment analysis accuracy is improved. The method could be enhanced by improving preprocessing models and considering additional context.

1. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Harnessing Ratings and Aspect-Sentiment to
Estimate Contradiction Intensity in
Temporal-Related Reviews
Ismail Badache - Sébastien Fournier - Adrian-Gabriel Chifu
Prénom.Nom@lsis.org
Laboratoire des Sciences de l’Information et des Systèmes
Aix-Marseille Université
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 1 / 24

2. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Plan
Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 2 / 24

3. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Introduction
• The diversity of opinions on a given topic ⇒ Contradiction
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 3 / 24

4. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Hypotheses
Hypothesis 1 : Reviews are related in time
Resource can be updated (e.g. corrected), and these updates will
be made after each session for the case of MOOCs (Massive Open
Online Courses) that are particularly the subject of our experiment.
After each session, users stop reviewing (silence) until the next
session. Therefore, temporal-related reviews mean the reviews
generated during a specific period (called in this paper : session).
Hypothesis 2 : Contradiction
A contradiction in reviews related to a web resource means
contradictory opinions expressed about a specific aspect, which is a
form of diversity of sentiments around the aspect for the same
resource.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 4 / 24

5. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Hypotheses
Hypothesis 3 : Contradiction intensity
An aspect with a negative sentiment in a review with a positive
rating (and vice-versa) has a more important impact on the
contradiction intensity than an aspect with a positive sentiment in
a review with a positive rating.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 5 / 24

6. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Research Questions
• How to identify a contradiction in reviews ?
• How to estimate contradiction intensity between reviews ?
• What is the impact of the joint consideration of polarity and
rating of the reviews on the measurement of the intensity of
contradiction ?
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 6 / 24

7. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Overview
Contradiction and Controversy Detection
• Wikipedia (Wang et al., 2014), News (Tsytsarau et al., 2014),
Debates analysis (Qiu et al., 2013) or generically on the Web
(Jang et Allan, 2016).
Aspects Detection
• Using HMM (Hidden Markov Models) or CRF (Conditional
Random Fields) as in (Hamdan et al., 2015).
• Unsupervised (Kim, 2013), Statistics rules (Poria, 2014).
Sentiment Analysis
• Lexicon (Turney, 2002) or Corpus (Mohammad et al., 2013).
• Naive Bayes (Pang et al., 2002), RNN (Socher et al., 2013).
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 7 / 24

8. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
How to detect contradiction ?
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 8 / 24

9. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Clustering reviews (Session)
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 9 / 24

10. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Identification of aspects
1 Terms frequency calculation of the reviews corpus,
2 Terms categorization (part-of-speech tagging) of reviews using
Stanford Parser 1,
3 Selection of terms having nominal category without
considering stopwords,
4 Selection of nouns with emotional terms in their
five-neighborhoods (using SentiWordNet 2 dictionary),
5 Extraction of the most frequent (used) terms in the corpus
among those selected in the previous step. These terms will be
considered as aspects.
1. http://nlp.stanford.edu:8080/parser/
2. http://sentiwordnet.isti.cnr.it/
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 10 / 24

11. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Identification of aspects : example
Step Description
(1)
course : 44219, material : 3286, assignments : 3118, content : 2947,
lecturer : 2705, ....... termei
(2)
The/DT lecturer/NN was/VBD an/DT annoying/VBG speaker/NN
and/CC very/RB repetitive/JJ ./. I/PRP found/VBD the/DT format-
ting/NN so/RB different/JJ from/IN other/JJ courses/NNS I/PRP
’ve/VBP taken/VBN ,/, that/IN it/PRP was/VBD hard/JJ to/TO
get/VB started/VBN and/CC figure/VB things/NNS out/RP ./.
(3) lecturer, speaker, formatting, things
(4) lecturer, speaker
(5) lecturer
The useful aspect is "lecturer"
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 11 / 24

12. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Detection of sentiment
Definition 1 : Sentiment
The sentiments are a real number in the range [−1, 1] which
indicates the polarity of the opinion expressed in the review segment
with respect to an aspect (called review-aspect ra). Negative and
positive values respectively represent negative and positive opinions.
Sentiment analysis model :
• Supervised model based on Naive Bayes.
• Negation handling (word preceded by "no", "not", "n’t").
• Intensifier and adverb processing (very, absolutely).
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 12 / 24

13. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Measure of contradiction
Definition 2 : Contradiction
There is a contradiction on an aspect between two segments of
reviews rai containing this aspect (ra1, ra2 ∈ D), where the
polarities around the aspect are opposite (pol(ra1) ∩ pol(ra2) = φ).
• pol(rai ) represents the function that returns the polarity
(positive, negative) of review-aspect rai .
Intensity of contradiction :
• Dimensions (poli , rati ) for each review-aspect rai .
• Dispersion of rai presented by a cloud of points.
• The greater the distance between the points rai with respect
to a centroid racentroid , the greater the degree of contradiction
is important.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 13 / 24

14. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Measure of contradiction
• Function of dispersion :
Disp(rapoli
rati
, D) =
1
n
n
i=1
Distance(poli , rati ) (1)
with :
Distance(poli , rati ) = (poli − pol)2 + (rati − rat)2 (2)
• Normalisation of ratings rati = rati −3
2 (rati ∈ [−1, 1]).
• Distance(poli , rati ) is the distance between the point rai of
the cloud and the centroid racentroid , and n is the number of
points rai of the cloud.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 14 / 24

15. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Measure of contradiction
Figure – Dispersion of reviews-aspect rai in the cloud (plane)
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 15 / 24

16. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Measure of contradiction
The coordinates (pol, rat) of the centroid racentroid can be
calculated in two different ways :
1 Centroid based on average of poli and rati
pol=
pol1+pol2+...+poln
n
; rat=
rat1+rat2+...+ratn
n
(3)
2 Centroid based on the weighted average of poli and rati
pol =
c1 · pol1 + c2 · pol2 + ... + cn · poln
n
(4)
rat =
c1 · rat1 + c2 · rat2 + ... + cn · ratn
n
(5)
with :
ci =
|poli − rati |
2n
(6)
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 16 / 24

17. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Objectives and dataset
• Evaluating the impact of sentiment analysis and rating on
contradiction detection in reviews around certain aspects.
• Evaluating the impact of the averaged and weighted centroid
on the contradiction intensity.
• Dataset : extracted from "coursera.org"
Field Total Number
Courses 2244
Courses Rated 1115
Reviews 73873
Ratings 298326
Reviews 1705
Reviews 1443
Reviews 3302
Reviews 12202
Reviews 55221
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 17 / 24

18. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Extracted aspects
• 22 aspects identified and extracted automatically from the
reviews of "coursera.org"
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 18 / 24

19. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
User Study
• Manual evaluation (contradictions and sentiments) :
• 3 assessors.
• 10 courses for each aspect.
• 1320 reviews/aspect of 220 courses.
• Kappa Cohen (contradictions) : k = 0.68.
• Kappa Cohen (sentiments) : k = 0.76.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 19 / 24

20. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Evaluation protocol
• Sentiments analyzer (Naive Bayes) :
• Training set : 50.000 reviews of IMDb 3
.
• Test set : the reviews-aspect of coursera.
• Accuracy : 79% (error rate 21%).
• Assessors’ judgments on sentiments are considered as perfect
(reference) results and represent an accuracy of 100%.
• Evaluation of the performance of our approach :
• Correlation study (official measure on SemEval tasks 4
).
• Using the correlation coefficients of Pearson and Spearman
between the contradiction judgments given by the assessors
and our obtained results.
3. http://ai.stanford.edu/~amaas/data/sentiment/
4. http://alt.qcri.org/semeval2016/task7/
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 20 / 24

21. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Results
Figure – Correlation between contradiction judgments and the results of
our approach (with sentiment analysis accuracy of 79%)
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 21 / 24

22. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Results
Figure – Correlation between contradiction judgments and the results of
our approach (with sentiment analysis accuracy of 100%)
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 22 / 24

23. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Conclusion
• Contribution : Estimating the intensity of contradiction
• Joint exploitation of polarities and ratings.
• Centroid calculation in 2 ways (averaged and weighted).
• limits :
• Dependence on the quality of sentiment analysis and aspect
detection models.
• Simplicity of pre-processing models.
• The sentences are not processed, only predefined window of 5
words before and after the aspect is considered.
• Perspectives :
• Improving the analysis of sentiments, aspects and sentences.
• Taking into account the profile of the user.
• Further scale-up experiments on other types of datasets are
also envisaged.
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 23 / 24

24. Introduction
Overview
Detection and intensity of contradiction
Experimental evaluation
Conclusion
Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 24 / 24