Presentation materials for ACM WebSci 2018 in Amsterdam, Netherlands on May 28th 2018.

1. Web Access Literacy Scale to Evaluate
How Critically Users Can Browse and
Search for Web Information
Yusuke Yamamoto1, Takehiro Yamamoto2,
Hiroaki Ohshima3, Hiroshi Kawakami2
1: Shizuoka University, Japan
2: Kyoto University, Japan
3: University of Hyogo, Japan
WebSci 2018: Session IIb

2. Background: Web information is not always correct
The number of medical Web sites
authorized by medical experts:
< 50%*
* E. Sillence et al., “Trust and Mistrust of Online Health Sites”, ACM CHI, pp.663-670, 2004

3. Possible approach in information science
Obtaining
correct information
(Semi-) automatic analysis
on information credibility

4. Examples of credibility analysis systems
*2 Y. Yamamoto and K. Tanaka. Enhancing Credibility Judgment of Web Search Results. In Proceedings of the 29th ACM SIGCHI Conference
on Human Factors in Computing Systems (CHI 2011), pages 1235–1244, 2011.
*1 Yin, X., Han, J., & Philip, S. Y. (2008). Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on
Knowledge and Data Engineering, 20(6), 796-808.
TruthFinder*1
Measures the consistency of a fact
describing an object
CowSearch*2
Provides credibility-related scores for
credibility judgment
These systems do not guarantee
the correctness of information
Limitation

5. Possible approach in information science
Enhancing user’s skills to
evaluate web information
Obtaining
correct information
(Semi-) automatic analysis
on information credibility

6. Research purpose
Development of scales and questionnaire to
self-evaluate the skills to “collect accurate
web information” Web access literacy (WAL)
1
Understanding of people’ strong points
and weak ones on web access literacy2

7. Web access literacy
Information literacy
Internet literacy
Media literacy
Skills to critically check web information and
collect accurate information from the web, using
information access system.
Web access literacy

8. Elements of Web access literacy
Critical thinking attitudes
Verification strategies
Skills to use web search engines
Tolerance for cognitive biases
KL : F A L : N: P KF: K :
2 P A PL K K : P L :K A K PL
7D L L KN N K L : NL L :K A L
K KF: LL:K K K CN F
. N L : N: KF: K : K K K:P
: NL K (
3 : : K : A N L LL C K :
8 K: K :L L PA :LL LL P KF: K )
: K F A :L L D L :L :
LN: : : :L
( 8 2NLNF : - R LD K : LD 8: D- 8A :L A 1NDNLA F: 0: A 4N :K : : LDS LD .: : L L ) ( (&&*
*3 G. Lindgaard et. al. An Exploration of Relations between Visual Appeal, Trustworthiness and Perceived Usability of Homepages. ACM TOCHI (2011)
*1 UC Berkeley Library. Evaluating resources. http://guides.lib.berkeley.edu/evaluating-resources.
,

9. Examples of question item candidates
Critical thinking attitudes (33)
Verification strategies (20)
Skills to use web search engines (5)
Tolerance for cognitive biases (11)
? - - 5 9 9
? - 9 ? - ? 5
? - ? 9 9- -
? - ? 9
5? -
- 95 - 9 5
A - ?
? 55 - - ? 9
- - ? 9
9 5 - ?

10. Online study
$
Development of web access literacy (WAL) scale
Analysis on the relation b/w WAL and external indicators
Purpose
Participant 534 05 30
Study method 32 0 0 25 ,280
, .85 / 5 8.2 1 08 2.0
Date 0 0 $ $
Reward ,8 2.2 ,
Language , , 0 0

11. Factor analysis results (#factor =7; promax rotation)Table 2: Factor analysis results of the web access literacy scale (promax rotation; N = 534). Items with an asterisk are reverse
code scale items.
Item M SD Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7
Factor 1: Logical approach (α = .88)
I am good at thinking about complicated problems in an organized way 3.25 0.96 .835 -.041 -.036 -.031 -.097 .024 .028
I am good at summarizing ideas 3.27 1.02 .754 -.067 .047 -.011 -.073 -.034 .046
I set a roadmap when thinking about something 3.70 0.91 . 705 .036 -.065 .025 .024 -.124 .045
I am good at making constructive proposals 3.18 0.94 .700 -.187 -.001 .015 .034 -.012 .152
I am good at explaining so that anyone can understand me 2.92 0.98 .685 -.188 .035 -.047 -.021 .103 .157
I am confident about my ability to think precisely 3.25 0.89 . 679 .023 -.012 -.076 -.025 -.035 .045
I get confused whenever thinking about complicated problems (*) 3.74 0.92 .620 .125 .056 -.051 -.034 -.016 -.184
I can maintain concentration while working on a problem 2.72 1.04 .600 -.095 -.037 .089 -.054 .078 -.031
I can continue to challenge tough problems 3.26 0.99 .508 .073 .204 -.017 -.094 -.006 -.010
I am easily distracted (*) 2.84 1.09 .484 -.077 -.122 .051 .109 -.035 -.113
I can look into a problem carefully 3.70 0.87 .480 .212 .044 -.011 .058 -.025 -.097
My colleagues often ask me to make judgments because I am fair 3.04 0.95 .415 -.062 .094 -.064 .206 .056 .100
Factor 2: Content-based verification strategy for information credibility (α = .87)
I try to spend as much time as possible on web searches 3.54 0.87 -.002 .804 .049 -.023 -.105 .011 -.175
I try to modify search queries to examine web page content more intricately 3.90 0.84 -.114 .741 .004 -.017 -.024 -.014 -.041
I try to compare multiple web pages 4.17 0.76 -.079 .741 .024 -.004 -.040 -.013 -.078
I try to issue multiple search queries to collect information in a broad perspective 3.85 0.86 -.069 .720 .067 -.027 .008 .001 -.128
I try to check other web pages or information resources to verify content credibility 3.73 0.93 -.012 .687 -.084 -.018 -.039 -.079 .160
I try to check similar or the same information on other web pages 3.54 0.90 -.042 .684 -.067 -.081 -.011 -.100 .143
I try to check to see that the information is complete and comprehensive 3.21 0.91 -.038 .492 .028 .024 -.014 .006 .263
I try to browse web pages in lower- and higher-ranked web search results 3.12 0.93 -.072 .490 -.021 .109 .016 .088 .066
I try to obtain evidence to verify the information on web pages 3.36 0.96 .017 .488 -.035 .071 -.046 -.011 .355
I try evaluating whether views represented on web pages are facts or opinions 3.79 0.98 .023 .428 -.002 .069 .086 -.147 .320
Factor 3: Inquisitiveness (α = .88)
I want to learn a lot by communicating with various types of people 3.84 1.02 -.052 -.027 .785 -.105 .003 -.018 .083
I want to learn various cultures 3.90 1.04 -.040 -.010 .724 .006 -.002 -.030 .055
I want to continue learning new things over my lifetime 4.07 0.94 .136 .027 .703 .048 -.052 -.058 .021
I want to learn as much as possible even if I am unsure if the information will be useful 3.81 1.05 .029 .029 .666 .087 .033 -.038 -.034
I am interested in people with opinions that differ from mine 3.76 1.00 -.089 -.082 .658 .013 .176 .060 .029
I like to challenge new things 3.67 1.07 .182 .051 .635 -.051 -.104 -.042 -.031
I want to learn more about any topic 3.62 1.05 .038 -.001 .615 .067 -.064 .092 -.053
I like to discuss with those who have opinions that differ from mine 3.44 1.11 -.053 -.113 .576 -.009 .178 .074 .074
Learning how foreigners think is useful 4.13 0.88 -.097 -.001 .570 -.083 .087 .000 .052
I try to ask about what I do not understand 4.01 0.93 .085 .169 .432 -.026 -.134 -.126 .098
Factor 4: Tolerance for biases in information credibility judgment (α = .81)
I trust information that is shared by many users or liked on social networking websites (*) 3.04 0.88 -.049 .063 -.071 .718 .067 -.104 .056
I trust information that my friends share on social networking websites (*) 2.96 0.85 .010 .034 -.071 .695 .071 -.148 .004
I trust information that my followers share on social networking websites (*) 3.35 0.85 .026 .082 -.025 .608 .041 -.171 .022
I trust information on easy-to-understand web pages (*) 2.51 0.71 .084 -.073 .016 .543 -.085 .036 .046
I trust posts that someone indicates are the best answers on Q/A sites (*) 2.80 0.93 -.016 .017 .020 .511 -.028 -.006 .043
I trust information on web pages with titles such as "the 20 best XXs" (*) 3.24 1.00 -.075 -.002 -.019 .507 .037 .084 .092
I trust information on web pages with a high rank on search engines (*) 2.66 0.81 -.068 .048 .002 .499 .014 .095 .068
I trust products or services with good reputations, on average, on review sites (*) 2.68 0.95 .071 -.118 -.027 .481 -.009 .068 -.003
I trust information on well-designed web pages (*) 2.77 0.68 .052 -.072 -.002 .474 -.027 .041 .038
Factor 5: Objectivity (α = .83)
I try to adopt an objective attitude when deciding something 3.81 0.87 .041 -.107 .051 -.070 .710 .053 .040
I try to make fair judgments 3.64 0.90 .022 -.054 .068 -.072 .691 -.010 -.048

12. 7 factors (1/2)
Logical approach (12)
* 3 3 ( ) 1 ) * (1 3 3 3 *
* 3 *
Content-based verification strategies (10)
3* ) (1 3 ( )
* ) 3 ( ) 3 3 3 ) 1
Inquisitiveness (10)
3 1 3 1 ( ) 3 ) 3 1
3 1 3 ) 1
Tolerance for biases in web credibility judgment (9)
3 3 * 1 * ( 3 3
3 3 3* 3
) * ) 1

13. 7 factors (2/2)
Objectivity (7)
4 4 4 - 4 -
4
Skill level in using web search engines (6)
4 1 4 4 4 4 4 - -
- 4 4 4 1 4 4
Author-based verification strategies (5)
4 - -4 1 4 4 1 4
4 - -4 4 -

14. Means of WAL score and factor scores
3.62
3.83
3.24
2.89
1.95
2.81
3.62
3.25
0 1 2 3 4 5
Objectivity
Inquisitiveness
Logical approach
Bias tolerance
Skill level using web search engine
Author-based verification stragegy
Content-based verification strategy
Web access literacy
Weak points are author-based verification
strategy, bias tolerance, and search engine skill

15. WAL score and learning activity (1/2)
No Yes
Experience of information literacy class
3.15
3.20
3.25
3.30
3.35
3.40
3.45
3.50
Webaccessliteracyscore
Not university-educated
University-educated
- 1 - 1 -1 -
1
1 -7 -
-7 -
Experience of information literacy classes had
positive impact on the WAL scores (F = 8.82, p < .01 )

16. WAL score and learning activity (2/2)
Educational background did not affect WAL scores
against our expectation (F=7.7x10-5, p=.993).
No Yes
Experience of information literacy class
3.15
3.20
3.25
3.30
3.35
3.40
3.45
3.50
Webaccessliteracyscore
Not university-educated
University-educated
8 - 1 - 1 -1 -
1
1 - -
- -

17. WAL score and experience in taking the literacy classes
0 1 2 3 4
Objectivity
Inquisitiveness
Logical approach
Bias tolerance
Skill level using web search engine
Author-based verification stragegy
Content-based verification strategy
with experience without experience
Any kinds of scores of participants with
the experience was greater than those without the
experience, except bias tolerance.

18. Implications for web access literacy (WAL) development
● People are likely to have weak points on author-
based verification strategy, bias tolerance, and
search engine skills
● Conventional literacy classes could contribute to
improve some factors on the WAL, although bias
tolerance could not be improved
● We need to re-design information literacy
classes or to develop credibility-oriented search
systems

19. Conclusion
Web access literacy scale to evaluate how
critically users can browse the web
2 3 2 3
-23 3 2 3
2 3 23 2
2 3 -
3 -
A complete list of questions is available
on our paper and my GitHub repository