1. Developing a multiple-document-
processing performance assessment
for epistemic literacy
Simon Knight
Karen Littleton
http://sjgknight.com
@sjgknight

2. Developing a multiple-document-
processing performance assessment
for epistemic literacy
Simon Knight
Karen Littleton
Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah,
Simon Buckingham Shum, Bart Rienties, Fridolin Wild
http://sjgknight.com
@sjgknight

3. Introduction
• New technologies:
– Foreground issues in
literacy – finding &
comprehending multi-
media
– Afford new opportunity
to explore & tackle these
issues

4. What is [Digital] literacy?
• [Digital] literacy –
– Comprehension in context of selecting & processing rich
online multimedia content with diverse sources (OECD,
2013)

6. Tasks
• Two collaborative tasks facilitated by a browser
addon
• One group provided with documents; the second
group searches on the web
• “A review of the best supported claims around
the risks” of a substance (herbicide or food
supplement)

7. 7
Friends of
the Earth:
Press
Release
(Urine
presence)
FoE
Commission
ed report
(‘scientific’)
(-ve)
Science-
Literacy
website:
Refutation
(+ve)
Farmer’s
Weekly
Reprints
(+ve)
Related
peer-review
publication
(Limited
risk)
Peer-review
publication
Health
danger
Reuters
Reprints
main claims
Blogger
Critiques
journal &
author
Peer-review
publication
(Limited
health risks)
Peer-review
review of
literature
(Limited risk
to health or
plants)
Peer-review
of lit
(Limited
risk; control
suggestions)
Urine
Health
Agricu
lture

8. 8
Statins
Citrinin
contamination
Concentration

9. Skills of literacy
Rouet [39] – students should be taught:
• Skill of integration: the ability to
connect prior and new information,
including across documents, and
including where claims are
inconsistent or contradictory
• Skill of sourcing: the ability to identify
parameters that characterize the
author and conditions of production
of the information
• Skill of corroboration: the ability to
check information against multiple
sources for its accuracy

10. Skills of literacy
• Communication – particularly meta-discourse
involving exploratory discussion of credentials,
sources, etc. – is key to collectively authored
written outputs
(Goldman and Scardamalia [17, p.260]).

11. Coagmento Tool
• Chat
• Foreground searches
• Share ‘snippets’
• Etherpad
• Tracks pageviews & copy
(/ctrl+c)

12. Epistemic Commitments
we should try
looking on
Wikipedia for
that

13. 13

14. A
C
B
¬A
B
………
………
…
14

15. A
C
By x (2002)
………
………
…
B
¬A
By y (2014)
B
By Gov (nd)
15

16. Source features
• Participants given 2 texts of varying quality are
more likely to favor high quality sources,
• BUT they make little reference to source
features (on average only 1.85 out of 10
features);
• and rarely (<6% of the 154 participants)
explicitly use source information for justifying
their evaluation of the text’s explanation
Kobayashi [26](2014)

17. Epistemic commitments
• Multiple Documents—Task-based Relevance
Assessment and Content Extraction (MD-TRACE)
model (Rouet/Rouet & Britt)
• 5 Steps:
1. Model construction
2. information need assessment;
3. document processing; - relevance, update needs,
seek new information
4. task product creation;
5. task product assessment.

18. “exploring students’ thought processes during online
searching allows examination of personal epistemology
not as a decontextualized set of beliefs, but as an
activated, situated aspect of cognition that influences the
knowledge construction process” (Hofer, 2004, p. 43).
The Lens of Epistemic Beliefs
Commitments

19. Certainty
19
A
C
By x
(2002)
……
……
……
…
Source
A B C
A
B
C
Simplicity
Justification
1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
2.The justification for knowledge – does knowledge come from observation, or from rules of
enquiry?
3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in
interaction with others?
4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute

20. Known variables (GPA, etc.)
• ISEQ
• GPA
• Self-report prior knowledge
• Self-report search skill
• Self-report partner familiarity

21. Trace constructs
• n Pages visited
• n Pages used
• Page-dwell >= 30 seconds
• n Pages visited/N pages visited
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-discourse
(authorship
Pub dates, etc.)
Topics
Other
chat
• n Pages used/N pages used
• Query distances
• Coded-chat instances (topic-terms,
source-terms, exploratory-terms)
• Collaborative-symmetry on <<&^^^

22. Outcome constructs
• 1-3 score on:
– Topic coverage
– Source diversity (largely ‘3’)
– Source quality/evaluation
– Synthesis
• 1-12 total score
22
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)

23. ProjectID
ISEQPaired
GPAPair
SearchSkillPair
PriorKnowlPair
PartnerFamilPair
SynthScore
TopicScore
SourDivScore
SourQualScore
TotatlScore
n-pages visited
n-pages used
n-dwelled
Chat-n-messages
Chat-n-explor
Chat-n-topic
Chat-n-source
queryDists (CIS ONLY)
Partner Symmetry
Outcome vars
Process vars
Control vars

24. 24
Predictors?
GPA, topic
knowledge,
search skill
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-discourse
(authorship
Pub dates, etc.)
Topics
Other
chat

25. ?
• ISEQ - psychometrics, etc.
• Outcome data (IRR)
• Trust
• Predictive value of summary data
(e.g. pages->Topics)
• Sequences
• Deep dive – key patterns, proof of concept of
those patterns. Critical incidents.

26. Thank you
Simon Knight
http://sjgknight.com
@sjgknight
Knight, S., Littleton, K., (2015). Developing a multiple-
document-processing performance assessment for epistemic
literacy. In The 5th International Learning Analytics &
Knowledge Conference (LAK15): Scaling Up: Big Data to Big
Impact, Poughkeepsie, NY. http://oro.open.ac.uk/41711/

27. 27
Predictors?
GPA, topic
knowledge,
search skill
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-data
(authorship
Pub dates, etc.)
Topics
Sources
(http://)
Other
chat

28. @sjgknight www.sjgknight.com
Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search,
Presented at the 2nd Society of the Query Conference, Amsterdam
• Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K.,
(2014). Epistemic Networks for Epistemic Commitments. Presented at the International
Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39254/
• Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology,
Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of
Learning Analytics. Available via The Open University Eprint Archive:
http://oro.open.ac.uk/39226 (HEA presentation version Blog/presentation)
• Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom
search engine tasks. Technology, Pedagogy and Education. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39181/
• Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R.
Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research
on Teaching Thinking. Routledge.
• In draft publications on computational approaches to discourse analysis + social account
of epistemic cognition

29. References
Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation
when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007
Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological
beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5
Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media
Institute: The Open University, UK. Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources:
Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027
Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of
the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern
research. Information Age Publishing (IAP).
Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning.
Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012
Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior
Research Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0
Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centered
design. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17).
Rowman & Littlefield.
Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online
information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001
Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information.Metacognition and
Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2
OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from
http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdf
Rouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge.
Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’
trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039
Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online
search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003
Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information?
Retrieved from http://dspace.ou.nl/handle/1820/4390
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional
solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030

30. Task1 (warmup)
Please type the answers to the following three prompts in your Task Pad (click in
the bar at the top of the browser). You may use the internet to find the answers.
• In 2010 what was the educational expenditure per primary student in
[XXX] as a % of GDP?
• In 2010 what was the total health expenditure as a % of GDP in [XXX]?
• How much (in US dollars) does a big mac cost in [ZZZ]?
Warmup – variables,
– XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana,
Guatemala, El Salvador, Ecuador (list of countries not too far from top of
alphabet, with data for 2010)
– Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada,
Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index)
If you find the warmup taking too long (over 10 minutes) but you feel you’re
now comfortable with using Coagmento, you should move on to Task 1.
• Task Submission Procedure [Slide 9]

31. MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate.
Your task is to act as an advisor to an official within the science ministry. You are advising an
official on the issues below. The official is not an expert in the area, but you can assume they are
a generally informed reader. They are interested in the best supported claims in the
documents. Produce a summary of the best supported claims you find and explain why you
think they are. Note you are not being asked to “create your own argument” or “summarise
everything you find” but rather, make a judgement about which claims have the strongest
support.
A colleague has already found a number of documents for you to process with your partner, you
should use these to extract the best supported claims (without using the internet to find further
material).
You should:
Read the questions/topic areas provided, these will require you to find information and
arguments in the documents to present the best supported of these, you should decide with
your partner which are best as you read.
Group information together by using headings in the Editor
You should work with your partner to explain why the claims you’ve found are the best available
You should spend about 45 minutes on this task
A review is coming up for the license of Glyphosate, the official would like to know the best
supported claims around its risks.
A colleague has collected some documents, available from the