This study was presented in the 20th International Conference on Web Information Systems Engineering (WISE 2019), 19-22 January 2020. ----- Fumiaki Saito, Yoshiyuki Shoji and Yusuke YamamotoHighlighting Weasel Sentences for Promoting Critical Information Seeking on the WebProceedings of the 20th International Conference on Web Information Systems Engineering (WISE 2019), pp.424-440, Hong Kong, China, November 2019.

1. Highlighting Weasel Sentences
for Promoting Critical
Information Seeking on the Web
Fumiaki Saito1 , Yoshiyuki Shoji2 ,Yusuke Yamamoto1
1:Shizuoka university, Japan
2: Aoyama Gakuin ,Japan January 19,2020
1
WISE 2019: Session S7

2. Background: Web information is not always correct
2
The number of medical Web sites
authorized by medical experts:
< 50%*
* E. Sillence et al., “Trust and Mistrust of Online Health Sites”, ACM CHI, pp.663-670, 2004

3. Possible approach in information science
3
Obtaining
correct information
(Semi-) automatic analysis
on information credibility

4. Examples of credibility analysis systems
*2 Y. Yamamoto and K. Tanaka. Enhancing Credibility Judgment of Web Search Results. In Proceedings of the 29th ACM SIGCHI Conference
on Human Factors in Computing Systems (CHI 2011), pages 1235–1244, 2011.
*1 Yin, X., Han, J., & Philip, S. Y. (2008). Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on
Knowledge and Data Engineering, 20(6), 796-808.
TruthFinder*1
Scores the consistency of fact
describing objects
CowSearch*2
Provides supporting information for
credibility judgment
The analysis does not guarantee
the correctness of information
Limitation
4

5. Possible approach in information science
5
Obtaining
correct information
(Semi-) automatic analysis
on information credibility
Careful examination on
information by users

6. 57%
Many people are not aware of Web information credibility!
82%
# of young Japanese people
who trust Web information *1
*1 Adobe Inc., “The State of Content : Rules of Engagement”, 2015
*2 S.Nakamura et al., “Trustworthiness analysis of Web search results (ECDL 2007)
Never mind,
never mind.
# of people who trust
information on SERP *2
I trust Google
6

7. Why don’t people often take care of Web information credibility
lTrust in search engine’s ranking1
lWrong metrics for quality judgment
(e.g. appearance of websites2)
lCognitive bias3
7
1:Kahneman, D.: Thinking, fast and slow, Macmillan (2011)
2:Pan, B., Hembrooke, H., Joachims, et al
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
3:Fogg, B. J, Soohoo, Cathy ,Danielson, David R, et al
How Do Users Evaluate the Credibility of Web Sites? A Study with over 2,500 Participants
♥

8. Research question
How can we promote
critical information seeking
when users do Web browsing?
8

9. Proposed system
9
Automatically detects and highlights
ambiguous sentences on the Web

10. Proposed system
10
Automatically detects and highlights
ambiguous sentences on the Web
They lack in evidences but seem to
have no problems in their claims
Ambiguous sentences
→
In this study

11. Proposed system to highlight ambiguous sentences
11
Makes users aware of information credibility
and promotes careful information seeking
!!

12. 12
Ambiguous sentence classifier1.

13. How to detect ambiguous
sentences?Q.
13
We focus on weasel expressionsA.

14. What are weasel expressions ?
14
l“Certain person concerned is ... ”
l“Research has shown...”
14
l“ It is often said ... ”
l“ It is widely thought ... ”
Who said that?
What is the truth?
Weasel Expressions create an impression that
something specific and meaningful has been said,
although their claim is ambiguous and lacks in evidence.

15. Classification of weasel sentences
We built a simple weasel sentence
classifier with Wikipedia data
15
Wikipedia
Ambiguous sentences
Non-ambiguous ones
Training/test set
ML
(SVM)

16. Training/test dataset to classify weasel sentences (1/2)
We focused on Wikipedia editting rules.
Some weasel expressions are annotated with the
special tags in Wikipedia.
Ex: Who said？
16
It is said that Company A
has said this hoax. [By whom?]

17. Weasel expressions in Wikipedia
l“The hoax is said to have been company
A, a media and advertising company that
wanted to show off their influence.[by whom?] ”
l“ ~ is an established theory, but
there are some objections. [who?]”
17
Image reference https://en.wikipedia.org/wiki/Wikipedia_logo

18. Training/test dataset to classify weasel sentences (2/2)
Positive examples (2236 senteneces)
Sentences annotated with [who?][by whom?] on
Wikipedia
18
Negative examples(2236 senteneces)
Sentences without [who?][by whom?] tags in
Wikipedia articles where weasel expressions
appear.

19. Features for classification
lBag-of-Words :nouns, verbs, and adjectives
lTypical example of ambiguous expression
listed in Wikipedia (27 phrases)
– "It is said that ..."
– "An expert argues that ..."
and so on
19

20. Classifier performance
Performance evaluation with
5-fold cross-validation
20
Accuracy Precision Recall F1
SVM(RBF) 0.764 0.772 0.748 0.760

21. Improvements for better classification (1/2)
21
We need to get more data from editing histories
on Wikipedia
Frequency of weasel annotation varies
depending on topics on Wikipedia
Topic categories of positive examples may
be biased
Improvements in dataset

22. Improvements for better classification (2/2)
22
Improvements in classification features
Because bag-of-words is simple as a feature,
features that consider surrounding sentences and
word order are required.

23. 23
Effect of our prototype
on user behaviors2.

24. How does our prototype affect users behaviors?
24
?
24
Can our prototype encourage users to
browse webpages more carefully?
If so, how do user behaviors change?
Q.

25. Hypothesis
lH1 The proposed system extends the time
expended in web information seeking.
lH2 The proposed system increases the
number of visited webpages.
lH3 The proposed system improves the users’
confidence in their decisions made through
web information seeking.
lH4 The above-mentioned effects vary with
users’ familiarity with the search topics.
25

26. User study
26
Participants were asked to search for
webpages to answer medical questions.
• Fixed search results:
Participants sought
100 fixed search results
• 4 search tasks:
We prepared 4 search tasks
about medical topics.
“Is cinnamon effective for diabetes?
Report your answer by Web search ”

27. Participant
The participants were recruited by crowdsourcing
and randomly assigned to two groups by UI
condition
lProposed group（105 participants ）
– The system highlighted the weasel sentences while
viewing the webpages
– Authors manually decided which sentence should be
highlighted as weasel sentences
– Participants learnt what highlighted sentences meant
before the task starts
lControlled group（83 participants ）
–Highlighting function was disabled
27

28. Focused behaviors
Measuring behavior on the web
–Session time
–SERP dwelling time
–Average dwelling time on one webpage
–Pageview (the number of visited webpages)
28

29. Questionnaire
lPre-questionnaire（6 levels）
– Degree of prior knowledge about search topic
– Expected answer
– Confidence about the answer
lPost questionnaire（6 levels）
– Answers for search task
– Confidence about the answers
29

30. Analysis method
lWe use Bayesian GLMM (generalized linear
mixture model) to model user behaviors
lFixed effects
– UI condition
– Topic familiarity
– Interaction between UI condition & topic familiarity
lRandom effects
– User
– Search topic
30

31. Session time
31
Participants with our prototype spent a
longer time in search tasks

32. Session time
If participants were more familiar with
search topic, there was greater difference
in session time due to UI conditions
32

33. Pageview
Participants with our prototype viewed
more webpages in search tasks
33

34. Pageview
The higher topic familiarity participants
had, the more webpages they viewed
34

35. Change of confidence
35
The interaction between UI and topic
familiarity affected confidence change.
If participants were not familiar with topics, our prototype
changed prior confidence more significantly.
If participants were familiar with topics, our prototype
changed prior confidence less significantly.

36. Discussion
lThe user study suggests that the
proposed system can enhance user
engagement in careful information
seeking on the web
– The system increased pageview and session time.
lWe need to investigate how user feel and
use highlighted sentences
– we didn’t understand why participants with our prototype
viewed more webpages and spent longer time in search
session
36

37. Conclusion
lDesigning a system to highlight weasel
sentences
– The classification performance of weasel sentences were
generally good performance but needs more improvement
lUser study to examine the system effect
– The proposed system can enhance user engagement in
critical information seeking on the web.
lFuture works
– Improvement of weasel classifier
– Investigating weasel sentences on the Web
37