1. EIGHTH GRADERS’ CRITICAL
EVALUATION OF WEB SOURCES
ABOUT A CONTROVERSIAL ISSUE
Angela K. Johnson
Ralph T. Putnam
Michigan State University
April 18, 2015
AERA Annual Conference
2. The
Information
Tornado
• Abundant
• Diverse
• Production
and
consumption
open to all
• Targeted
• Purposes
obscured
http://www.illustrationsource.com/stock/image/47836/technology-tornado/?&results_per_page=1&detail=TRUE&page=54
3.
4. Purpose Statement
How do 8th grade students
approach the task of critically
evaluating the trustworthiness of
web sources presenting opposing
views on a controversial issue?
6. New Literacy Theory
• Online reading demands skills above and
beyond offline reading.
• Evaluation—more difficult.
• Authorship—most often overlooked.
(Braten, Stromso, Britt 2009; Coiro 2007; Leu, Kinzer, Coiro, &
Cammack, D. W. 2004)
7. Argument/Persuasion Theory
• Argument = intratextual challenges
• Prior opinion bias:
• more open to supporting evidence
• more critical of refuting evidence
• Prior opinion must be
separated from task of
evaluating source
Prior Opinion
Site Evalution
8. Challenges of
Evaluating Internet
Sites about
Controversies
Online
Processing
Complexity
Multiple
Text
Processing
Complexity
Argumentative
Text Processing
Complexity
Bracketing
Prior
Opinion =
Complexity
Evaluation =
Complex
Higher Order
Thinking
9. Research Questions
1. Can 8th grade students effectively rate the
trustworthiness of sites about a
controversial issue?
2. Do their prior opinions on the issue affect
their ability to rate trustworthiness?
1. What criteria do they use to determine
trustworthiness?
1. Do stronger and weaker readers differ with
regard to any of the above?
10. Evaluation Task
• Topic: Should school personnel should be
allowed to possess concealed weapons in
schools?
• Rate the trustworthiness of 5 sources:
High Trust Pro Site High Trust Con Site
Low Trust Pro Site Low Trust Con Site
Neutral Site
11. High Quality Con Site linkhere
Well known
source
Authors fully
identified
12. High Quality Con Site linkhere
Credentials of authors
clearly written below
article, and those
credentials showed
relevant expertise (a
public policy professor
at the University of
Chicago and
13. High Quality Con Site linkhere
Also listed all sources
in a bibliography at
bottom of article
14. Low Quality Con Site linkhere
Lesser known site
clearly identified as a
blog
Author
identified only
as “karoli” with
no credentials
15. Low Quality Con Site linkhere
No sources
cited anywhere
on the page
16. High Quality Pro Site linkhereWell known
source, formal
publication
Clearly identifies
author
17. High Quality Pro Site linkhere
Author’s
credentials listed
at bottom of page,
identified as a
“former member of
U.S. Sentencing
Commission” and
author of a book
on the topic
Sources clearly
listed at bottom of
article
19. Low Quality Pro Site linkhere
Author identified
only by name,
“dr.” and
“ministry”
Neither “dr.” nor
“minister” shows
expertise on
topic of guns in
schools
20. Low Quality Pro Site linkhere
No sources cited
at bottom of
page.
22. Average Site Ratings by Reading Ability
2.00
2.50
3.00
3.50
4.00
4.50
low high
Site Quality
low readers
ave readers
high readers
23. Do students’ prior opinions affect their
ability to rate trustworthiness?
24. Average Site Ratings by Student Opinion
2.00
2.50
3.00
3.50
4.00
4.50
Con Guns Pro GunsSite Stance
Con
Neutral
Pro
25. Top Criteria by Reading Level (ave times cited/reader)
Low Readers Average Readers High Readers
1 reasoning/commentary
(2.60 )
evidence/facts/data (2.53) evidence/facts/data (3.48)
2 evidence/facts/data (2.30) reasoning/commentary
(2.34)
reasoning/commentary (2.83)
3 justifies stance (1.80) justifies stance (1.69) balance/bias (2.17)
4 logical/makes sense (1.25) logical/makes sense
(1.66)
source/author info available (1.97)
5 length/amount/detail of
info (1.20)
info relevance (1.03) logical/makes sense (1.34)
6 info relevance (1.0) agree/disagree (1.0) quotes other sources (.83)
7 organization/rhetoric (.85) focused (.94) graphs/charts/pictures (.69)
8 balance/bias (.80) graphs/charts/pics (.81) focused (.66)
published/vetted (.66)
9 focused (.80) balance/bias (.78) length/amount/detail of info (.62)
refers to spec source/type (.62)
number of authors/sources (.62)10 records random fact (.65) language (.72)
26. Top Criteria by Reading Level (ave times cited/reader)
Low Readers Average Readers High Readers
1 reasoning/commentary
(2.60 )
evidence/facts/data (2.53) evidence/facts/data (3.48)
2 evidence/facts/data (2.30) reasoning/commentary
(2.34)
reasoning/commentary (2.83)
3 justifies stance (1.80) justifies stance (1.69) balance/bias (2.17)
4 logical/makes sense (1.25) logical/makes sense
(1.66)
source/author info available (1.97)
5 length/amount/detail of
info (1.20)
info relevance (1.03) logical/makes sense (1.34)
6 info relevance (1.0) agree/disagree (1.0) quotes other sources (.83)
7 organization/rhetoric (.85) focused (.94) graphs/charts/pictures (.69)
8 balance/bias (.80) graphs/charts/pics (.81) focused (.66)
published/vetted (.66)
9 focused (.80) balance/bias (.78) length/amount/detail of info (.62)
refers to spec source/type (.62)
number of authors/sources (.62)10 records random fact (.65) language (.72)
Both LOW and AVE
readers frequently lost
sight of task goal by
justifying a STANCE ON
THE ISSUE rather than
justifying their
EVALUATION…
Or by recording
RANDOM FACTS
irrelevant to the task.
27. Top Criteria by Reading Level (ave times cited/reader)
Low Readers Average Readers High Readers
1 reasoning/commentary
(2.60 )
evidence/facts/data (2.53) evidence/facts/data (3.48)
2 evidence/facts/data (2.30) reasoning/commentary
(2.34)
reasoning/commentary (2.83)
3 justifies stance (1.80) justifies stance (1.69) balance/bias (2.17)
4 logical/makes sense (1.25) logical/makes sense
(1.66)
source/author info available (1.97)
5 length/amount/detail of
info (1.20)
info relevance (1.03) logical/makes sense (1.34)
6 info relevance (1.0) agree/disagree (1.0) quotes other sources (.83)
7 organization/rhetoric (.85) focused (.94) graphs/charts/pictures (.69)
8 balance/bias (.80) graphs/charts/pics (.81) focused (.66)
published/vetted (.66)
9 focused (.80) balance/bias (.78) length/amount/detail of info (.62)
refers to spec source/type (.62)
number of authors/sources (.62)10 records random fact (.65) language (.72)
Average readers = also
less able to separate
their personal opinion
from site evaluation.
28. Top Criteria by Reading Level (ave times cited/reader)
Low Readers Average Readers High Readers
1 reasoning/commentary
(2.60 )
evidence/facts/data (2.53) evidence/facts/data (3.48)
2 evidence/facts/data (2.30) reasoning/commentary
(2.34)
reasoning/commentary (2.83)
3 justifies stance (1.80) justifies stance (1.69) balance/bias (2.17)
4 logical/makes sense (1.25) logical/makes sense
(1.66)
source/author info available (1.97)
5 length/amount/detail of
info (1.20)
info relevance (1.03) logical/makes sense (1.34)
6 info relevance (1.0) agree/disagree (1.0) quotes other sources (.83)
7 organization/rhetoric (.85) focused (.94) graphs/charts/pictures (.69)
8 balance/bias (.80) graphs/charts/pics (.81) focused (.66)
published/vetted (.66)
9 focused (.80) balance/bias (.78) length/amount/detail of info (.62)
refers to spec source/type (.62)
number of authors/sources (.62)10 records random fact (.65) language (.72)
High readers more
focused on
AUTHORSHIP and
SOURCING
29. Implications for Research/Instruction
• What scaffolds would best help low and average
readers mediate cognitive load?
• Teach students to attend to authorship and sourcing,
especially average and low readers?
• Is CCSS moving students in right direction?
End
Good afternoon. My name is Angela Johnson and I’m a doctoral candidate at Michigan State University. The study I’ll be presenting today is a look at Eighth Graders’ Critical Evaluation of Web Sources About a Controversial Issue.
As most of us are aware, we live in an age in which information is abundant and diverse in quality, in which barriers to participation and consumption are low, and in which messages are increasingly targeted toward specific users for specific purposes. In this tornado of incoming information, users need to know how to determine what is trustworthy and what is not.
As we know, the Internet has all kinds of sites with all kinds of purposes. It is entirely likely that students might come across a site like this one while doing research on the health effects of fast food. Now, those of us who recognize The Onion, know this is a spoof site. But our students come to the Internet with less world experience than we have… and not all Internet messages are this obvious.
SO…This research was designed to zero in on how 8th graders evaluate trustworthiness of Internet sites that present information about controversial issues, in this case the issue of whether school personnel should be allowed to carry concealed weapons on school grounds.
There are several theories relevant to this study. The first of these is CFT, which tells us that when people are doing complex tasks in ill-structured domains, they need to practice those tasks in authentic environments, without too much oversimplification. This is because oversimplification doesn’t transfer well to ill-structured environments. The internet is ill-structured, and evaluation is a complex thing to do. So giving students the opportunity to practice it authentically is essential.
NLT relevant b/c it tells us that our kids are going to have more trouble online, but evaluation/consideration of authorship are particularly important to constructing meaning
P relevant b/c it adds another layer of complexity, requiring “bracking” of biases in order to evaluate/consider authorship effectively
M relebant b/c it’s the one tool kids are going to need to grapple with all of these things: here, specifically : argument, source, “bracketing”
The second theory grounding this study is NLT, which posits that online reading is not isomophic with offline reading. Online reading requires in some cases more and in some cases different skills than offline reading. For example, evaluation is more important and more difficult online. Authorship is more important and often more difficult to ascertain than when one is doing offline reading.
Also grounding this study are theories of persuasion and argumentative text processing. Processing written arguments is more challenging that most expository or narrative text because of its complex organization. In addition, prior opinion is known to lead to bias in the assimilation of new information, because people tend to be more critical of evidence refuting their opinion and more accepting of evidence supporting it. If one is to evaluate a site fairly, one must, to some extent, set aside these biases to view the site with some measure of objectivity.
In summary, the task presents multiple complexities:
Online processing complexity greater than offline.
Multiple texts also present greater complexity b/c reader must synthesize opposing ideas and source models into a broader content model.
Argumentative text processing is more demanding than narrative or expository text.
Prior opinion further complicates evaluation of argumentative text.
Finally, evaluation is itself a higher order thinking process that requires metacognitive skill.
These are the research questions posed by the present study.
The task asked students to imagine they were doing online research to inform themselves about this topic because their school board was considering allowing employees to carry concealed weapons. We chose two pro and two con sides, one of each of those was high and one was low quality. We also added a neutral site presenting both sides, but this was omitted from statistical tests.
Published, vetted source
Author identified by name
Unknown source
Author identified only by screen name: karoli
No author credentials
Sources are not cited
Published, vetted source
Author identified
Author credentials on page
Sources are cited
Unknown source
Author identified by name only
Author credentials not relevant to topic
minister and dr. (of what?).
Site provides no sourcing for the information presented.
Students at all levels recognized difference between low and high quality sites. On average, students at all levels rated more trustworthy sites higher than less trustworthy sites.
However, this was influence by reading level: the higher the reading level, the more clearly students differentiated. BLUE = highest level readers; GREEN = lowest level readers
There was a positive effect of student opinion on site ratings. So, students tended to rate sites that they agreed with higher than sites they disagreed with. Note, however, that ratings for pro sites were on average higher than ratings for con sites. This was due to an anomaly with one site in particular: the low quality pro site (the site by the dr/minister). That average rating of that site was a full point HIGHER than the high quality con site.
Look first at the BLUE line: these are PRO STANCE STUDENTS. As expected, they rate the pro sites MUCH higher than the con sites.
Then look at the NEUTRAL STANCE STUDENTS in orange. They also rate the pro sites higher, though the difference is a bit less.
Finally, look at the CON STANCE STUDENTS in green. Surprisingly, they also rate the pro sites higher, though only by a tiny bit.
This is because ALL students found the dr./minister site much more trustworthy than it deserved.
So, imagine that a correction for the inflated rating of that one site is made by dragging all of the points down a bit. NOW the lines are what you’d expect.
Notice that ALL levels of students most often cited REASONING/COMMENTARY and FACTS/EVIDENCE as criteria for evaluation. These are #1 and #2 most cited criteria by all students.
However, from here the criteria vary more by level. Notice now what the remaining top criteria include at BOTH low and average reading levels. Here, the RED indicates criteria that signal the reader is having some difficulty. BOTH low and ave readers frequently lost sight of task goal (evaluation). Frequently lapsed into justifying a stance on the topic rather than justifying their evaluation of the sites OR recording random facts.
Notice now the remaining top criteria at BOTH low and average reading levels. BOTH low and ave readers frequently lost sight of task goal (evaluation). Frequently lapsed into justifying a stance on the topic rather than justifying their evaluation of the sites or recorded random facts that suggested their thinking was off task.
High readers did not have the same problem in remaining focused on the task of source evaluation (no red criteria). They also focused much more on authorship and sourcing, looking for source and author information, considering whether source was published/vetted, and looking for specific types of more reliable sources. Neither low or average readers did either of these. IN SUM these suggest that the task is especially taxing for even average readers = HIGH COGNITIVE LOAD?
--Still a VERY complex task with a high cognitive load = What scaffolds can be provided to help low and average readers mediate the cognitive load and retain focus on task goal?
--SPECIFICALLY, would scaffolds helping students focus more on authorship and sourcing shift focus from their own prior opinion to the perspective of the author and the site itself?
--Could emphasis on evidence and reasoning the CCSS be helping our kids to focus on those when reading texts presenting opinions on controversial issues?
