ORCAs: Online Research and Comprehension Assessments developed at the New Literacies Lab, University of Connecticut

1. Developing Reliable, Valid, and
Practical Assessments of Online
Research and Comprehension
(ORCAs)
Donald J. Leu, Ph.D
The New Literacies Research Lab
The Neag School
University of Connecticut

2. Today


Introduce a performance based assessment
of online research and comprehension in
science.
Share the results of several studies of online
research and comprehension.

3. Institute of Educational Science - U.S. Department of Education
Principal Investigator
Donald J. Leu, The University of Connecticut
Co-Principal Investigators
Jonna Kulikowich, The Pennsylvania State University
Nell Sedransk, National Institute of Statistical Sciences
Julie Coiro, University of Rhode Island
Scientific Advisory Board
P. David Pearson, The University of California, Berkeley
Irwin Kirsch, Educational Testing Service
Rand Spiro, Michigan State University
Elizabeth Stage, Lawrence Hall of Science, Berkeley
Glenn Kleimann, Friday Institute, NCSU
3

4. Our Focus Is The New Literacies of
Online Research and
Comprehension

Defines how we read online when we
conduct informal and formal research to
learn and develop new knowledge.




Locate
Evaluate
Synthesize
Communicate
Castek, 2008; Coiro & Dobler, 2007;
Henry, 2007; Leu, Castek, Hartman,
Coiro, Henry, Kulikowich, & Lyver, 2005;
Leu, O’Byrne, Zawilinski, McVerry,
& Everett-Cacopardo, 2009; Leu, Kinzer,
Coiro, Castek, & Henry, 2013

5. Locate
Can the student locate the correct email message in an
inbox on the first click?
Can the student use appropriate keywords in a search
engine?
Can the student locate the best site for a task from a set of
search engine results on the first click?
Can the student locate and communicate the correct
website addresses from two different search tasks.

6. Evaluate
Can the student identify the author of the website?
Can the student evaluate the author's level of
expertise?
Can the student identify the author's point of view?
Can the student evaluate the reliability of a
website?

7. Synthesize
Can students provide a summary of one important element from
the first website?
Can students use their own words to integrate one detail from
each of the first two websites?
Can students use their own words to integrate one detail from
each of the second two websites?
Can students use their own words to develop an argument after
reading all four websites?

8. Communicate: Email (Wiki)
Does the student include the correct email address in an email
message?
Does the student include an appropriate subject line in an
email message?
Does the student include an appropriate greeting in an email
message to an important, unfamiliar person?
Does the student compose and send a well-structured, short
report of their research in an email with sources and
appropriate argument structure, containing at least one
relevant claim and at least two pieces of evidence.

9. Two Formats:
Closed - Multiple Choice
Four Research Questions
TOPIC
TYPE OF RESEARCH COMMUNICATION TOOL
PROBLEM
FOR THE REPORT
How do energy drinks affect Learn more about
heart health?
email
How can snacks be hearthealthy?
Learn more about
email
Do cosmetic contact lenses
harm your eyes?
Investigate conflicting
claims
Wiki
Do videogames harm your
eyesight?
Investigate conflicting
claims
Wiki

10. Reliability and Validity Data



Final statistics are being calculated
These eight ORCAs were the best
performing from 24 tested the previous
year.
Items revised based on the previous
year’s data.

11. ORCA-Closed

12. 12

13. 13

14. 14

15. 15

16. 16

17. 17

18. 18

19. Reading to
Locate
Information

20. Reading to
Locate
Information
Online

21. Reading to Synthesize
Information Online

22. Reading to Critically Evaluate
Information Online

23. Reading to Critically Evaluate
Information Online

24. Reading to Critically Evaluate
Information Online

25. Reading to Critically Evaluate
Information Online

26. Reading to Critically Evaluate
Information Online

27. Reading to Critically Evaluate
Information Online

28. Reading to Critically Evaluate
Information Online

29. Reading to Critically Evaluate
Information Online

30. Reading to Critically Evaluate
Information Online

31. Reading to Critically Evaluate
Information Online

32. Reading to Critically Evaluate
Information Online

33. Reading and Writing to
Communicate Information Online:
Email Task
33

34. ORCA-Multiple Choice: An
Identical Sequence of Items

35. How Well Do Students Conduct Online
Research and Comprehend?
Students are only minimally prepared.
Average % of Correct
Responses
Locate Skills
52.9%
Evaluate Skills
54.3%
Synthesis Skills
64.6%
Communicate Skills
44.4%
TOTAL
54%

36. Skill
Reading to Locate Online Information
% Correct
52.9%
Locate the correct email message in an inbox or the
correct section of a wiki.
60.5%
Use appropriate key words in a search engine.
59.5%
Locate the correct site in a set of search engine results.
56.0%
Locate and share correct website addresses in two
different search tasks.
33.0%
Reading to Evaluate Online Information
Identify the author of a website.
Evaluate an author’s level of expertise.
Identify an author’s point of view.
Evaluate the reliability of a website.
54.3%
78.5%
47.5%
51.0%
39.5%

37. Reading to Synthesize Online Information
64.6%
Summarize an important element from one website.
77.5%
Synthesize important elements from two websites.
63.5%
Synthesize important elements from a second set of two
57.0%
websites.
Synthesize important elements from websites to develop an
60.5%
argument.
Writing to Communicate Online Information
44.4%
Include the correct address in email and make a wiki entry
in the correct location.
41.5%
Include an appropriate subject line in email and an
appropriate heading in a wiki.
53.0%
Include an appropriate greeting in email and use descriptive
voice in a wiki.
49.5%
Compose a short report of research, including sources, in
email and wiki.
33.0%

38. A Second Study: Students in
Rich and Poor School Districts
Leu, D. J., Coiro, J., O’Byrne, W. I.,
Zawilinski, L., McVerry, J. G.,
Cacopardo, H., Kennedy, C., &
Forzani, E. (2011). Online
reading comprehension
assessment (ORCA): A
preliminary study of online
reading comprehension ability
in rich and poor school districts.
Paper presented at the meeting
of the Literacy Research
Association. Jacksonville, FL.

39. In the U.S., The Rich Get
Richer and the Poor Get Poorer
in OFFLINE READINGReading. 90/10 Income Achievement & BlackOffline COMPREHENSION:
Average Difference in S.D. Units on National Assessments
White Gaps
in Reading, 1943-2001 Cohorts
*Adapted from: Reardon, S.F. (2011). The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither Opportunity? Rising Inequality and the Uncertain Life Chances of Low-Income Children. New York: Russell Sage Foundation
Press.
Reardon, S.F. (2011). The widening academic achievement gap between the rich and the poor: New
evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither Opportunity?
Rising Inequality and the Uncertain Life Chances of Low-Income Children. New York: Russell Sage
Foundation Press.

40. Our Study’s School District
Differences
West Town
East Town
Median Family Income
$119,338
$58,981
% of Families Below Poverty
Line
2.1%
11.8%
% of Students Eligible for
4%
Free/Reduced Price Lunches
67%

41. A Significant Achievement Gap
Existed in Offline Reading
(State Test: Reading)
282.6
215.1
t (237) = 14.34 p = .000


West Town (Rich) Mean = 282.6 (SD = 41.54)

East Town (Poor) Mean = 215.1 (SD = 31.07)
eta squared = .466 (large)

42. A Significant Achievement Gap Existed in
Online Research and Comprehension
Abilities
15.00
7.65
t (255) = 9.80, p = .000

West Town Mean = 15.00 (SD=5.69)

East Town Mean = 7.65 (SD=4.39)

eta squared = .319 (large)

43. ...Even When an ANCOVA Analysis
Was Conducted
Differences Controlled: Offline Reading + Prior
Knowledge
12.96
9.27
F (1,234) = 15.84, p = .001

West Town (rich) adjusted mean = 12.96

partial eta squared = .063

East Town (poor) adjusted mean = 10.27

(medium)

44. A Third Study: Comparing Online
Research Skills in Two States
(n=1,129)
State 1
State 2
(laptops)
TOTAL
Locate (8)
4.52
4.64
4.58
Evaluate (8)
3.61
3.32
3.47
Synthesize (8)
6.07
5.86
5.97
Communicate (8)
4.22
4.00
4.11
TOTAL (32)
18.42
17.81
18.13
State 1: 4th in Median Family Income/ Few Laptops
State 2: 32nd in Median Family Income/1 to 1 Laptops

45. Adjusted Total Mean Scores
When Covariates of SES and
Prior Knowledge Controlled
State 1
Adjusted Means
State 2
(laptops)
17.56
19.08*
*p < .05 F (1, 1021) = 14.854, p = .000
45

46. A Practical Assessment Of
Online Research and
Comprehension

47. The Design of Our Assessments:
Practicality Concerns

Administration time?



Multiple Choice - 25 minutes
Closed - 40 minutes
Scoring?



Closed - Auto scoring and hand scoring
Format?


Multiple Choice - Auto scoring
Multiple Choice and Performance Based
Reports?

Specific skill performance (16 skills)

Skill Area performance (4 Areas)

Total score
47

48. Invitations:

Please send me names, titles, emails of
any superintendent, principal, or middle
school teacher whom you think may be
interested in viewing a video and
completing a 15 minute survey.
donald.leu@uconn.edu

49. Developing Reliable, Valid, and
Practical Assessments of Online
Research and Comprehension
(ORCAs)
Donald J. Leu, Ph.D
The New Literacies Research Lab
The Neag School
University of Connecticut