Developing a multiple-document-processing performance assessment for epistemic literacy

Simon Knight
Simon KnightLecturer at UTS: University of Technology Sydney
Developing a multiple-document-
processing performance assessment
for epistemic literacy
Simon Knight
Karen Littleton
http://sjgknight.com
@sjgknight
Developing a multiple-document-
processing performance assessment
for epistemic literacy
Simon Knight
Karen Littleton
Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah,
Simon Buckingham Shum, Bart Rienties, Fridolin Wild
http://sjgknight.com
@sjgknight
Introduction
• New technologies:
– Foreground issues in
literacy – finding &
comprehending multi-
media
– Afford new opportunity
to explore & tackle these
issues
What is [Digital] literacy?
• [Digital] literacy –
– Comprehension in context of selecting & processing rich
online multimedia content with diverse sources (OECD,
2013)
 Developing a multiple-document-processing performance assessment for epistemic literacy
Tasks
• Two collaborative tasks facilitated by a browser
addon
• One group provided with documents; the second
group searches on the web
• “A review of the best supported claims around
the risks” of a substance (herbicide or food
supplement)
7
Friends of
the Earth:
Press
Release
(Urine
presence)
FoE
Commission
ed report
(‘scientific’)
(-ve)
Science-
Literacy
website:
Refutation
(+ve)
Farmer’s
Weekly
Reprints
(+ve)
Related
peer-review
publication
(Limited
risk)
Peer-review
publication
Health
danger
Reuters
Reprints
main claims
Blogger
Critiques
journal &
author
Peer-review
publication
(Limited
health risks)
Peer-review
review of
literature
(Limited risk
to health or
plants)
Peer-review
of lit
(Limited
risk; control
suggestions)
Urine
Health
Agricu
lture
8
Statins
Citrinin
contamination
Concentration
Skills of literacy
Rouet [39] – students should be taught:
• Skill of integration: the ability to
connect prior and new information,
including across documents, and
including where claims are
inconsistent or contradictory
• Skill of sourcing: the ability to identify
parameters that characterize the
author and conditions of production
of the information
• Skill of corroboration: the ability to
check information against multiple
sources for its accuracy
Skills of literacy
• Communication – particularly meta-discourse
involving exploratory discussion of credentials,
sources, etc. – is key to collectively authored
written outputs
(Goldman and Scardamalia [17, p.260]).
Coagmento Tool
• Chat
• Foreground searches
• Share ‘snippets’
• Etherpad
• Tracks pageviews & copy
(/ctrl+c)
Epistemic Commitments
we should try
looking on
Wikipedia for
that
13
A
C
B
¬A
B
………
………
…
14
A
C
By x (2002)
………
………
…
B
¬A
By y (2014)
B
By Gov (nd)
15
Source features
• Participants given 2 texts of varying quality are
more likely to favor high quality sources,
• BUT they make little reference to source
features (on average only 1.85 out of 10
features);
• and rarely (<6% of the 154 participants)
explicitly use source information for justifying
their evaluation of the text’s explanation
Kobayashi [26](2014)
Epistemic commitments
• Multiple Documents—Task-based Relevance
Assessment and Content Extraction (MD-TRACE)
model (Rouet/Rouet & Britt)
• 5 Steps:
1. Model construction
2. information need assessment;
3. document processing; - relevance, update needs,
seek new information
4. task product creation;
5. task product assessment.
“exploring students’ thought processes during online
searching allows examination of personal epistemology
not as a decontextualized set of beliefs, but as an
activated, situated aspect of cognition that influences the
knowledge construction process” (Hofer, 2004, p. 43).
The Lens of Epistemic Beliefs
Commitments
Certainty
19
A
C
By x
(2002)
……
……
……
…
Source
A B C
A
B
C
Simplicity
Justification
1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
2.The justification for knowledge – does knowledge come from observation, or from rules of
enquiry?
3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in
interaction with others?
4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute
Known variables (GPA, etc.)
• ISEQ
• GPA
• Self-report prior knowledge
• Self-report search skill
• Self-report partner familiarity
Trace constructs
• n Pages visited
• n Pages used
• Page-dwell >= 30 seconds
• n Pages visited/N pages visited
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-discourse
(authorship
Pub dates, etc.)
Topics
Other
chat
• n Pages used/N pages used
• Query distances
• Coded-chat instances (topic-terms,
source-terms, exploratory-terms)
• Collaborative-symmetry on <<&^^^
Outcome constructs
• 1-3 score on:
– Topic coverage
– Source diversity (largely ‘3’)
– Source quality/evaluation
– Synthesis
• 1-12 total score
22
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)
ProjectID
ISEQPaired
GPAPair
SearchSkillPair
PriorKnowlPair
PartnerFamilPair
SynthScore
TopicScore
SourDivScore
SourQualScore
TotatlScore
n-pages visited
n-pages used
n-dwelled
Chat-n-messages
Chat-n-explor
Chat-n-topic
Chat-n-source
queryDists (CIS ONLY)
Partner Symmetry
Outcome vars
Process vars
Control vars
24
Predictors?
GPA, topic
knowledge,
search skill
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-discourse
(authorship
Pub dates, etc.)
Topics
Other
chat
?
• ISEQ - psychometrics, etc.
• Outcome data (IRR)
• Trust
• Predictive value of summary data
(e.g. pages->Topics)
• Sequences
• Deep dive – key patterns, proof of concept of
those patterns. Critical incidents.
Thank you
Simon Knight
http://sjgknight.com
@sjgknight
Knight, S., Littleton, K., (2015). Developing a multiple-
document-processing performance assessment for epistemic
literacy. In The 5th International Learning Analytics &
Knowledge Conference (LAK15): Scaling Up: Big Data to Big
Impact, Poughkeepsie, NY. http://oro.open.ac.uk/41711/
27
Predictors?
GPA, topic
knowledge,
search skill
Source
Diversity
Source
Quality/
Evaluation
Synthesis Topic
coverageOutcome:
(Peer assess?)
Pages
visited
Pages
used
Collaborative symmetry
Query diversity
Chat data
Exploratory/
epistemic
Meta-data
(authorship
Pub dates, etc.)
Topics
Sources
(http://)
Other
chat
@sjgknight www.sjgknight.com
Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search,
Presented at the 2nd Society of the Query Conference, Amsterdam
• Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K.,
(2014). Epistemic Networks for Epistemic Commitments. Presented at the International
Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39254/
• Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology,
Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of
Learning Analytics. Available via The Open University Eprint Archive:
http://oro.open.ac.uk/39226 (HEA presentation version Blog/presentation)
• Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom
search engine tasks. Technology, Pedagogy and Education. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39181/
• Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R.
Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research
on Teaching Thinking. Routledge.
• In draft publications on computational approaches to discourse analysis + social account
of epistemic cognition
References
Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation
when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007
Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological
beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5
Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media
Institute: The Open University, UK. Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources:
Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027
Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of
the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern
research. Information Age Publishing (IAP).
Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning.
Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012
Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior
Research Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0
Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centered
design. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17).
Rowman & Littlefield.
Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online
information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001
Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information.Metacognition and
Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2
OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from
http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdf
Rouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge.
Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’
trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039
Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online
search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003
Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information?
Retrieved from http://dspace.ou.nl/handle/1820/4390
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional
solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030
Task1 (warmup)
Please type the answers to the following three prompts in your Task Pad (click in
the bar at the top of the browser). You may use the internet to find the answers.
• In 2010 what was the educational expenditure per primary student in
[XXX] as a % of GDP?
• In 2010 what was the total health expenditure as a % of GDP in [XXX]?
• How much (in US dollars) does a big mac cost in [ZZZ]?
Warmup – variables,
– XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana,
Guatemala, El Salvador, Ecuador (list of countries not too far from top of
alphabet, with data for 2010)
– Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada,
Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index)
If you find the warmup taking too long (over 10 minutes) but you feel you’re
now comfortable with using Coagmento, you should move on to Task 1.
• Task Submission Procedure [Slide 9]
MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate.
Your task is to act as an advisor to an official within the science ministry. You are advising an
official on the issues below. The official is not an expert in the area, but you can assume they are
a generally informed reader. They are interested in the best supported claims in the
documents. Produce a summary of the best supported claims you find and explain why you
think they are. Note you are not being asked to “create your own argument” or “summarise
everything you find” but rather, make a judgement about which claims have the strongest
support.
A colleague has already found a number of documents for you to process with your partner, you
should use these to extract the best supported claims (without using the internet to find further
material).
You should:
Read the questions/topic areas provided, these will require you to find information and
arguments in the documents to present the best supported of these, you should decide with
your partner which are best as you read.
Group information together by using headings in the Editor
You should work with your partner to explain why the claims you’ve found are the best available
You should spend about 45 minutes on this task
A review is coming up for the license of Glyphosate, the official would like to know the best
supported claims around its risks.
A colleague has collected some documents, available from the
1 of 31

Recommended

Learning Analytics in Medical Education by
Learning Analytics in Medical EducationLearning Analytics in Medical Education
Learning Analytics in Medical EducationAbelardo Pardo
1.7K views25 slides
edmedia2014-learning-analytics-keynote by
edmedia2014-learning-analytics-keynoteedmedia2014-learning-analytics-keynote
edmedia2014-learning-analytics-keynoteSimon Buckingham Shum
7K views92 slides
Using data (analytics and analysis) to guide (e)teaching and (e)learning (upd... by
Using data (analytics and analysis) to guide (e)teaching and (e)learning (upd...Using data (analytics and analysis) to guide (e)teaching and (e)learning (upd...
Using data (analytics and analysis) to guide (e)teaching and (e)learning (upd...Poh-Sun Goh
1.9K views1 slide
Xiao Hu "Overview of the Space of Learning Analytics and Educational Data Min... by
Xiao Hu "Overview of the Space of Learning Analytics and Educational Data Min...Xiao Hu "Overview of the Space of Learning Analytics and Educational Data Min...
Xiao Hu "Overview of the Space of Learning Analytics and Educational Data Min...CITE
1.3K views14 slides
Introduction to Learning Analytics - Framework and Implementation Concerns by
Introduction to Learning Analytics - Framework and Implementation ConcernsIntroduction to Learning Analytics - Framework and Implementation Concerns
Introduction to Learning Analytics - Framework and Implementation ConcernsTore Hoel
1.3K views67 slides
OLT conference Learning analytics by
OLT conference Learning analyticsOLT conference Learning analytics
OLT conference Learning analyticsShirley Alexander
2.2K views29 slides

More Related Content

What's hot

Learning analytics: An opportunity for higher education? by
Learning analytics: An opportunity for higher education?Learning analytics: An opportunity for higher education?
Learning analytics: An opportunity for higher education?Dragan Gasevic
2.5K views50 slides
Requirements for Learning Analytics by
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning AnalyticsTore Hoel
1.6K views45 slides
Using student data to inform support, pedagogy & curricula: ethical issues & ... by
Using student data to inform support, pedagogy & curricula: ethical issues & ...Using student data to inform support, pedagogy & curricula: ethical issues & ...
Using student data to inform support, pedagogy & curricula: ethical issues & ...University of South Africa (Unisa)
2K views41 slides
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey by
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, TurkeyAn insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkeystrehlst
1.4K views13 slides
Learning analytics in higher education: Promising practices and lessons learned by
Learning analytics in higher education: Promising practices and lessons learnedLearning analytics in higher education: Promising practices and lessons learned
Learning analytics in higher education: Promising practices and lessons learnedBodong Chen
739 views39 slides
Our Learning Analytics are Our Pedagogy by
Our Learning Analytics are Our PedagogyOur Learning Analytics are Our Pedagogy
Our Learning Analytics are Our PedagogySimon Buckingham Shum
5.3K views85 slides

What's hot(20)

Learning analytics: An opportunity for higher education? by Dragan Gasevic
Learning analytics: An opportunity for higher education?Learning analytics: An opportunity for higher education?
Learning analytics: An opportunity for higher education?
Dragan Gasevic2.5K views
Requirements for Learning Analytics by Tore Hoel
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning Analytics
Tore Hoel1.6K views
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey by strehlst
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, TurkeyAn insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
strehlst1.4K views
Learning analytics in higher education: Promising practices and lessons learned by Bodong Chen
Learning analytics in higher education: Promising practices and lessons learnedLearning analytics in higher education: Promising practices and lessons learned
Learning analytics in higher education: Promising practices and lessons learned
Bodong Chen739 views
Learning analytics are more than a technology by Dragan Gasevic
Learning analytics are more than a technologyLearning analytics are more than a technology
Learning analytics are more than a technology
Dragan Gasevic3K views
Open Learning Analytics panel at Open Education Conference 2014 by Stian Håklev
Open Learning Analytics panel at Open Education Conference 2014Open Learning Analytics panel at Open Education Conference 2014
Open Learning Analytics panel at Open Education Conference 2014
Stian Håklev2K views
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching by Rafael Scapin, Ph.D.
Learning Analytics in Education:  Using Student’s Big Data to Improve TeachingLearning Analytics in Education:  Using Student’s Big Data to Improve Teaching
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
Open Science and Ethics studies in SLE research by davinia.hl
Open Science and Ethics studies in SLE researchOpen Science and Ethics studies in SLE research
Open Science and Ethics studies in SLE research
davinia.hl316 views
Gobert, Dede, Martin, Rose "Panel: Learning Analytics and Learning Sciences" by CITE
Gobert, Dede, Martin, Rose "Panel: Learning Analytics and Learning Sciences"Gobert, Dede, Martin, Rose "Panel: Learning Analytics and Learning Sciences"
Gobert, Dede, Martin, Rose "Panel: Learning Analytics and Learning Sciences"
CITE1.1K views
State and Directions of Learning Analytics Adoption (Second edition) by Dragan Gasevic
State and Directions of Learning Analytics Adoption (Second edition)State and Directions of Learning Analytics Adoption (Second edition)
State and Directions of Learning Analytics Adoption (Second edition)
Dragan Gasevic2.1K views
Technologies to support self-directed learning through social interaction by Dragan Gasevic
Technologies to support self-directed learning through social interactionTechnologies to support self-directed learning through social interaction
Technologies to support self-directed learning through social interaction
Dragan Gasevic1.8K views
Learning analytics are more than measurement by Dragan Gasevic
Learning analytics are more than measurementLearning analytics are more than measurement
Learning analytics are more than measurement
Dragan Gasevic2.9K views
Learning Analytics – Ethical questions and dilemmas by Tore Hoel
Learning Analytics  – Ethical questions and dilemmasLearning Analytics  – Ethical questions and dilemmas
Learning Analytics – Ethical questions and dilemmas
Tore Hoel2.4K views
A Learning Analytics Approach by MehrnooshV
A Learning Analytics ApproachA Learning Analytics Approach
A Learning Analytics Approach
MehrnooshV1.8K views
Wearable technologies should promote adaptive learners by Dragan Gasevic
Wearable technologies should promote adaptive learnersWearable technologies should promote adaptive learners
Wearable technologies should promote adaptive learners
Dragan Gasevic1.4K views

Viewers also liked

Competency-based Medical Education by
Competency-based Medical EducationCompetency-based Medical Education
Competency-based Medical EducationJanet Corral
9.3K views33 slides
Learning and Training for Transfer by
Learning and Training for TransferLearning and Training for Transfer
Learning and Training for TransferPoh-Sun Goh
11.2K views1 slide
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest... by
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...Richard Oelmann
4.2K views30 slides
One (more) representation of the continuum of learning by
One (more) representation of the continuum of learningOne (more) representation of the continuum of learning
One (more) representation of the continuum of learningPoh-Sun Goh
8.6K views1 slide
David Williamson Shaffer: Epistemic Games (Paris juin 2011) by
David Williamson Shaffer: Epistemic Games (Paris juin 2011)David Williamson Shaffer: Epistemic Games (Paris juin 2011)
David Williamson Shaffer: Epistemic Games (Paris juin 2011)Thomas Constant
1K views119 slides
Introduction to Competency-based Education by
Introduction to Competency-based EducationIntroduction to Competency-based Education
Introduction to Competency-based EducationD2L
6.9K views13 slides

Viewers also liked(20)

Competency-based Medical Education by Janet Corral
Competency-based Medical EducationCompetency-based Medical Education
Competency-based Medical Education
Janet Corral9.3K views
Learning and Training for Transfer by Poh-Sun Goh
Learning and Training for TransferLearning and Training for Transfer
Learning and Training for Transfer
Poh-Sun Goh11.2K views
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest... by Richard Oelmann
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...
Learning Outcomes and Assessment - Achieving Constructive Alignment Treforest...
Richard Oelmann4.2K views
One (more) representation of the continuum of learning by Poh-Sun Goh
One (more) representation of the continuum of learningOne (more) representation of the continuum of learning
One (more) representation of the continuum of learning
Poh-Sun Goh8.6K views
David Williamson Shaffer: Epistemic Games (Paris juin 2011) by Thomas Constant
David Williamson Shaffer: Epistemic Games (Paris juin 2011)David Williamson Shaffer: Epistemic Games (Paris juin 2011)
David Williamson Shaffer: Epistemic Games (Paris juin 2011)
Thomas Constant1K views
Introduction to Competency-based Education by D2L
Introduction to Competency-based EducationIntroduction to Competency-based Education
Introduction to Competency-based Education
D2L6.9K views
Lak 05 by dws1d
Lak 05Lak 05
Lak 05
dws1d2.5K views
The Value of Competency-based Medical Education Across the Continuum by MedCouncilCan
The Value of Competency-based Medical Education Across the ContinuumThe Value of Competency-based Medical Education Across the Continuum
The Value of Competency-based Medical Education Across the Continuum
MedCouncilCan5K views
CBME and Assessment by jakinyi
CBME and Assessment CBME and Assessment
CBME and Assessment
jakinyi5.7K views
Digital educational resource hyperlinked index vs restaurant menu by Poh-Sun Goh
Digital educational resource hyperlinked index vs restaurant menuDigital educational resource hyperlinked index vs restaurant menu
Digital educational resource hyperlinked index vs restaurant menu
Poh-Sun Goh16.5K views
Personal and personalised learning and teaching (updated) by Poh-Sun Goh
Personal and personalised learning and teaching (updated)Personal and personalised learning and teaching (updated)
Personal and personalised learning and teaching (updated)
Poh-Sun Goh3.7K views
Using data (analytics:analysis) to guide (e)teaching and (e)learning by Poh-Sun Goh
Using data (analytics:analysis) to guide (e)teaching and (e)learningUsing data (analytics:analysis) to guide (e)teaching and (e)learning
Using data (analytics:analysis) to guide (e)teaching and (e)learning
Poh-Sun Goh3.1K views
Learning Analytics - George Siemens by Sungjin Nam
Learning Analytics - George SiemensLearning Analytics - George Siemens
Learning Analytics - George Siemens
Sungjin Nam4.6K views
Applied learning analytics to class and program learning activities by Poh-Sun Goh
Applied learning analytics to class and program learning activitiesApplied learning analytics to class and program learning activities
Applied learning analytics to class and program learning activities
Poh-Sun Goh4.2K views
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient... by Simon Buckingham Shum
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient...XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient...
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient...
Simon Buckingham Shum10.2K views
Reflect on how Bloom's taxonomy, Miller's pyramid and the Kirkpatrick model m... by Poh-Sun Goh
Reflect on how Bloom's taxonomy, Miller's pyramid and the Kirkpatrick model m...Reflect on how Bloom's taxonomy, Miller's pyramid and the Kirkpatrick model m...
Reflect on how Bloom's taxonomy, Miller's pyramid and the Kirkpatrick model m...
Poh-Sun Goh27.4K views
Personal Learning Graph (PLeG) by Dragan Gasevic
Personal Learning Graph (PLeG)Personal Learning Graph (PLeG)
Personal Learning Graph (PLeG)
Dragan Gasevic30.5K views
Dcla13 discourse, computation and context – sociocultural dcla by Simon Knight
Dcla13 discourse, computation and context – sociocultural dclaDcla13 discourse, computation and context – sociocultural dcla
Dcla13 discourse, computation and context – sociocultural dcla
Simon Knight3.1K views
The Evidence Hub: Harnessing the Collective Intelligence of Communities to Bu... by Anna De Liddo
The Evidence Hub: Harnessing the Collective Intelligence of Communities to Bu...The Evidence Hub: Harnessing the Collective Intelligence of Communities to Bu...
The Evidence Hub: Harnessing the Collective Intelligence of Communities to Bu...
Anna De Liddo6.4K views
Personal and personalised learning and teaching by Poh-Sun Goh
Personal and personalised learning and teachingPersonal and personalised learning and teaching
Personal and personalised learning and teaching
Poh-Sun Goh3.7K views

Similar to Developing a multiple-document-processing performance assessment for epistemic literacy

Information Literacy and the 2.0s by
Information Literacy and the 2.0sInformation Literacy and the 2.0s
Information Literacy and the 2.0sKara Jones
632 views17 slides
Embracing Imperfection in Learning Analytics by
Embracing Imperfection in Learning AnalyticsEmbracing Imperfection in Learning Analytics
Embracing Imperfection in Learning AnalyticsSimon Buckingham Shum
1.5K views43 slides
Analíticas del aprendizaje: una perspectiva crítica by
Analíticas del aprendizaje: una perspectiva críticaAnalíticas del aprendizaje: una perspectiva crítica
Analíticas del aprendizaje: una perspectiva críticaCENT
9.2K views71 slides
Social media as a tool for researchers by
Social media as a tool for researchersSocial media as a tool for researchers
Social media as a tool for researchersJari Laru
349 views37 slides
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal... by
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...Simon Buckingham Shum
2.4K views50 slides
Borgman orcid dryadsymposiumoxford20130523 by
Borgman orcid dryadsymposiumoxford20130523Borgman orcid dryadsymposiumoxford20130523
Borgman orcid dryadsymposiumoxford20130523ORCID, Inc
1.3K views10 slides

Similar to Developing a multiple-document-processing performance assessment for epistemic literacy(20)

Information Literacy and the 2.0s by Kara Jones
Information Literacy and the 2.0sInformation Literacy and the 2.0s
Information Literacy and the 2.0s
Kara Jones632 views
Analíticas del aprendizaje: una perspectiva crítica by CENT
Analíticas del aprendizaje: una perspectiva críticaAnalíticas del aprendizaje: una perspectiva crítica
Analíticas del aprendizaje: una perspectiva crítica
CENT9.2K views
Social media as a tool for researchers by Jari Laru
Social media as a tool for researchersSocial media as a tool for researchers
Social media as a tool for researchers
Jari Laru349 views
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal... by Simon Buckingham Shum
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
Borgman orcid dryadsymposiumoxford20130523 by ORCID, Inc
Borgman orcid dryadsymposiumoxford20130523Borgman orcid dryadsymposiumoxford20130523
Borgman orcid dryadsymposiumoxford20130523
ORCID, Inc1.3K views
Learning Analytics - A New Discipline and Bits of Semantics by Dragan Gasevic
Learning Analytics - A New Discipline and Bits of SemanticsLearning Analytics - A New Discipline and Bits of Semantics
Learning Analytics - A New Discipline and Bits of Semantics
Dragan Gasevic4.6K views
Use Online Journal Club Events To Train The Educational Research Literacy: Re... by Jingjing Lin
Use Online Journal Club Events To Train The Educational Research Literacy: Re...Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Jingjing Lin2 views
Thinking About the Making of Data by Paul Groth
Thinking About the Making of DataThinking About the Making of Data
Thinking About the Making of Data
Paul Groth441 views
Designing a Survey Study to Measure the Diversity of Digital Learners by IJITE
Designing a Survey Study to Measure the Diversity of Digital Learners Designing a Survey Study to Measure the Diversity of Digital Learners
Designing a Survey Study to Measure the Diversity of Digital Learners
IJITE18 views
Designing a Survey Study to Measure the Diversity of Digital Learners by IJITE
Designing a Survey Study to Measure the Diversity of Digital LearnersDesigning a Survey Study to Measure the Diversity of Digital Learners
Designing a Survey Study to Measure the Diversity of Digital Learners
IJITE9 views
Semantic Technologies in Learning Environments by Dragan Gasevic
Semantic Technologies in Learning EnvironmentsSemantic Technologies in Learning Environments
Semantic Technologies in Learning Environments
Dragan Gasevic1.7K views
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区 by Jingjing Lin
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
Jingjing Lin3 views
Building capacity for open, data-driven science - Grand Rounds by Kaitlin Thaney
Building capacity for open, data-driven science - Grand RoundsBuilding capacity for open, data-driven science - Grand Rounds
Building capacity for open, data-driven science - Grand Rounds
Kaitlin Thaney2.1K views
Inma leonard become a library detective by ileonardgalan
Inma leonard become a library detectiveInma leonard become a library detective
Inma leonard become a library detective
ileonardgalan338 views
Let the Trumpet Sound by Johan Koren
Let the Trumpet SoundLet the Trumpet Sound
Let the Trumpet Sound
Johan Koren909 views

More from Simon Knight

CiteLearn: An academic tool for learning to cite sources by
CiteLearn: An academic tool for learning to cite sourcesCiteLearn: An academic tool for learning to cite sources
CiteLearn: An academic tool for learning to cite sourcesSimon Knight
177 views9 slides
Aligning Learning Analytics with Classroom Practices & Needs by
Aligning Learning Analytics with Classroom Practices & NeedsAligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & NeedsSimon Knight
492 views88 slides
Calibrating Assessment Literacy Through Benchmarking Tasks by
  Calibrating Assessment Literacy Through Benchmarking Tasks  Calibrating Assessment Literacy Through Benchmarking Tasks
Calibrating Assessment Literacy Through Benchmarking TasksSimon Knight
351 views26 slides
Who to believe: How epistemic cognition can inform science communication (key... by
Who to believe: How epistemic cognition can inform science communication (key...Who to believe: How epistemic cognition can inform science communication (key...
Who to believe: How epistemic cognition can inform science communication (key...Simon Knight
511 views45 slides
Writing Analytics for Epistemic Features of Student Writing #icls2016 talk by
Writing Analytics for Epistemic Features of Student Writing #icls2016 talkWriting Analytics for Epistemic Features of Student Writing #icls2016 talk
Writing Analytics for Epistemic Features of Student Writing #icls2016 talkSimon Knight
1.7K views63 slides
Wikimedia UK volunteer structure proposals by
Wikimedia UK volunteer structure proposalsWikimedia UK volunteer structure proposals
Wikimedia UK volunteer structure proposalsSimon Knight
537 views13 slides

More from Simon Knight(15)

CiteLearn: An academic tool for learning to cite sources by Simon Knight
CiteLearn: An academic tool for learning to cite sourcesCiteLearn: An academic tool for learning to cite sources
CiteLearn: An academic tool for learning to cite sources
Simon Knight177 views
Aligning Learning Analytics with Classroom Practices & Needs by Simon Knight
Aligning Learning Analytics with Classroom Practices & NeedsAligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & Needs
Simon Knight492 views
Calibrating Assessment Literacy Through Benchmarking Tasks by Simon Knight
  Calibrating Assessment Literacy Through Benchmarking Tasks  Calibrating Assessment Literacy Through Benchmarking Tasks
Calibrating Assessment Literacy Through Benchmarking Tasks
Simon Knight351 views
Who to believe: How epistemic cognition can inform science communication (key... by Simon Knight
Who to believe: How epistemic cognition can inform science communication (key...Who to believe: How epistemic cognition can inform science communication (key...
Who to believe: How epistemic cognition can inform science communication (key...
Simon Knight511 views
Writing Analytics for Epistemic Features of Student Writing #icls2016 talk by Simon Knight
Writing Analytics for Epistemic Features of Student Writing #icls2016 talkWriting Analytics for Epistemic Features of Student Writing #icls2016 talk
Writing Analytics for Epistemic Features of Student Writing #icls2016 talk
Simon Knight1.7K views
Wikimedia UK volunteer structure proposals by Simon Knight
Wikimedia UK volunteer structure proposalsWikimedia UK volunteer structure proposals
Wikimedia UK volunteer structure proposals
Simon Knight537 views
Epistemic networks for Epistemic Commitments by Simon Knight
Epistemic networks for Epistemic CommitmentsEpistemic networks for Epistemic Commitments
Epistemic networks for Epistemic Commitments
Simon Knight1.4K views
Learning analytics for learning and becoming in practice - #ICLS2014 workshop by Simon Knight
Learning analytics for learning and becoming in practice - #ICLS2014 workshopLearning analytics for learning and becoming in practice - #ICLS2014 workshop
Learning analytics for learning and becoming in practice - #ICLS2014 workshop
Simon Knight851 views
Wikimedia Strategy - making it impactful, measuring impact, and thinking abou... by Simon Knight
Wikimedia Strategy - making it impactful, measuring impact, and thinking abou...Wikimedia Strategy - making it impactful, measuring impact, and thinking abou...
Wikimedia Strategy - making it impactful, measuring impact, and thinking abou...
Simon Knight2.7K views
Learning and analytics – where do the two meet? #HEABigData summit day by Simon Knight
Learning and analytics – where do the two meet? #HEABigData summit dayLearning and analytics – where do the two meet? #HEABigData summit day
Learning and analytics – where do the two meet? #HEABigData summit day
Simon Knight2.2K views
Finding Knowledge: Assessing Knowledge in the Age of Search by Simon Knight
Finding Knowledge: Assessing Knowledge in the Age of SearchFinding Knowledge: Assessing Knowledge in the Age of Search
Finding Knowledge: Assessing Knowledge in the Age of Search
Simon Knight1.2K views
Analysing Learning Through Mediawiki by Simon Knight
Analysing Learning Through MediawikiAnalysing Learning Through Mediawiki
Analysing Learning Through Mediawiki
Simon Knight958 views
Mediawiki for oer and learning analytics - understanding learning resources a... by Simon Knight
Mediawiki for oer and learning analytics - understanding learning resources a...Mediawiki for oer and learning analytics - understanding learning resources a...
Mediawiki for oer and learning analytics - understanding learning resources a...
Simon Knight1.8K views
LAK13: Epistemology, Pedagogy, Assessment and Learning Analytics by Simon Knight
LAK13: Epistemology, Pedagogy, Assessment and Learning AnalyticsLAK13: Epistemology, Pedagogy, Assessment and Learning Analytics
LAK13: Epistemology, Pedagogy, Assessment and Learning Analytics
Simon Knight1.4K views
SoLAR Storm talk: epistemology, pedagogy, assessment by Simon Knight
SoLAR Storm talk: epistemology, pedagogy, assessmentSoLAR Storm talk: epistemology, pedagogy, assessment
SoLAR Storm talk: epistemology, pedagogy, assessment
Simon Knight2.3K views

Recently uploaded

Volf work.pdf by
Volf work.pdfVolf work.pdf
Volf work.pdfMariaKenney3
89 views43 slides
BUSINESS ETHICS MODULE 1 UNIT I_A.pdf by
BUSINESS ETHICS MODULE 1 UNIT I_A.pdfBUSINESS ETHICS MODULE 1 UNIT I_A.pdf
BUSINESS ETHICS MODULE 1 UNIT I_A.pdfDr Vijay Vishwakarma
40 views25 slides
INT-244 Topic 6b Confucianism by
INT-244 Topic 6b ConfucianismINT-244 Topic 6b Confucianism
INT-244 Topic 6b ConfucianismS Meyer
45 views77 slides
JQUERY.pdf by
JQUERY.pdfJQUERY.pdf
JQUERY.pdfArthyR3
105 views22 slides
Narration lesson plan by
Narration lesson planNarration lesson plan
Narration lesson planTARIQ KHAN
75 views11 slides
JRN 362 - Lecture Twenty-Three (Epilogue) by
JRN 362 - Lecture Twenty-Three (Epilogue)JRN 362 - Lecture Twenty-Three (Epilogue)
JRN 362 - Lecture Twenty-Three (Epilogue)Rich Hanley
41 views57 slides

Recently uploaded(20)

INT-244 Topic 6b Confucianism by S Meyer
INT-244 Topic 6b ConfucianismINT-244 Topic 6b Confucianism
INT-244 Topic 6b Confucianism
S Meyer45 views
JQUERY.pdf by ArthyR3
JQUERY.pdfJQUERY.pdf
JQUERY.pdf
ArthyR3105 views
Narration lesson plan by TARIQ KHAN
Narration lesson planNarration lesson plan
Narration lesson plan
TARIQ KHAN75 views
JRN 362 - Lecture Twenty-Three (Epilogue) by Rich Hanley
JRN 362 - Lecture Twenty-Three (Epilogue)JRN 362 - Lecture Twenty-Three (Epilogue)
JRN 362 - Lecture Twenty-Three (Epilogue)
Rich Hanley41 views
Create a Structure in VBNet.pptx by Breach_P
Create a Structure in VBNet.pptxCreate a Structure in VBNet.pptx
Create a Structure in VBNet.pptx
Breach_P86 views
The Future of Micro-credentials: Is Small Really Beautiful? by Mark Brown
The Future of Micro-credentials:  Is Small Really Beautiful?The Future of Micro-credentials:  Is Small Really Beautiful?
The Future of Micro-credentials: Is Small Really Beautiful?
Mark Brown75 views
Six Sigma Concept by Sahil Srivastava.pptx by Sahil Srivastava
Six Sigma Concept by Sahil Srivastava.pptxSix Sigma Concept by Sahil Srivastava.pptx
Six Sigma Concept by Sahil Srivastava.pptx
Sahil Srivastava44 views
12.5.23 Poverty and Precarity.pptx by mary850239
12.5.23 Poverty and Precarity.pptx12.5.23 Poverty and Precarity.pptx
12.5.23 Poverty and Precarity.pptx
mary850239381 views
Retail Store Scavenger Hunt.pptx by jmurphy154
Retail Store Scavenger Hunt.pptxRetail Store Scavenger Hunt.pptx
Retail Store Scavenger Hunt.pptx
jmurphy15452 views
The Accursed House by Émile Gaboriau by DivyaSheta
The Accursed House  by Émile GaboriauThe Accursed House  by Émile Gaboriau
The Accursed House by Émile Gaboriau
DivyaSheta251 views
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE... by Nguyen Thanh Tu Collection
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...

Developing a multiple-document-processing performance assessment for epistemic literacy

  • 1. Developing a multiple-document- processing performance assessment for epistemic literacy Simon Knight Karen Littleton http://sjgknight.com @sjgknight
  • 2. Developing a multiple-document- processing performance assessment for epistemic literacy Simon Knight Karen Littleton Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah, Simon Buckingham Shum, Bart Rienties, Fridolin Wild http://sjgknight.com @sjgknight
  • 3. Introduction • New technologies: – Foreground issues in literacy – finding & comprehending multi- media – Afford new opportunity to explore & tackle these issues
  • 4. What is [Digital] literacy? • [Digital] literacy – – Comprehension in context of selecting & processing rich online multimedia content with diverse sources (OECD, 2013)
  • 6. Tasks • Two collaborative tasks facilitated by a browser addon • One group provided with documents; the second group searches on the web • “A review of the best supported claims around the risks” of a substance (herbicide or food supplement)
  • 7. 7 Friends of the Earth: Press Release (Urine presence) FoE Commission ed report (‘scientific’) (-ve) Science- Literacy website: Refutation (+ve) Farmer’s Weekly Reprints (+ve) Related peer-review publication (Limited risk) Peer-review publication Health danger Reuters Reprints main claims Blogger Critiques journal & author Peer-review publication (Limited health risks) Peer-review review of literature (Limited risk to health or plants) Peer-review of lit (Limited risk; control suggestions) Urine Health Agricu lture
  • 9. Skills of literacy Rouet [39] – students should be taught: • Skill of integration: the ability to connect prior and new information, including across documents, and including where claims are inconsistent or contradictory • Skill of sourcing: the ability to identify parameters that characterize the author and conditions of production of the information • Skill of corroboration: the ability to check information against multiple sources for its accuracy
  • 10. Skills of literacy • Communication – particularly meta-discourse involving exploratory discussion of credentials, sources, etc. – is key to collectively authored written outputs (Goldman and Scardamalia [17, p.260]).
  • 11. Coagmento Tool • Chat • Foreground searches • Share ‘snippets’ • Etherpad • Tracks pageviews & copy (/ctrl+c)
  • 12. Epistemic Commitments we should try looking on Wikipedia for that
  • 13. 13
  • 16. Source features • Participants given 2 texts of varying quality are more likely to favor high quality sources, • BUT they make little reference to source features (on average only 1.85 out of 10 features); • and rarely (<6% of the 154 participants) explicitly use source information for justifying their evaluation of the text’s explanation Kobayashi [26](2014)
  • 17. Epistemic commitments • Multiple Documents—Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet/Rouet & Britt) • 5 Steps: 1. Model construction 2. information need assessment; 3. document processing; - relevance, update needs, seek new information 4. task product creation; 5. task product assessment.
  • 18. “exploring students’ thought processes during online searching allows examination of personal epistemology not as a decontextualized set of beliefs, but as an activated, situated aspect of cognition that influences the knowledge construction process” (Hofer, 2004, p. 43). The Lens of Epistemic Beliefs Commitments
  • 19. Certainty 19 A C By x (2002) …… …… …… … Source A B C A B C Simplicity Justification 1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected? 2.The justification for knowledge – does knowledge come from observation, or from rules of enquiry? 3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others? 4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute
  • 20. Known variables (GPA, etc.) • ISEQ • GPA • Self-report prior knowledge • Self-report search skill • Self-report partner familiarity
  • 21. Trace constructs • n Pages visited • n Pages used • Page-dwell >= 30 seconds • n Pages visited/N pages visited Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-discourse (authorship Pub dates, etc.) Topics Other chat • n Pages used/N pages used • Query distances • Coded-chat instances (topic-terms, source-terms, exploratory-terms) • Collaborative-symmetry on <<&^^^
  • 22. Outcome constructs • 1-3 score on: – Topic coverage – Source diversity (largely ‘3’) – Source quality/evaluation – Synthesis • 1-12 total score 22 Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?)
  • 24. 24 Predictors? GPA, topic knowledge, search skill Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?) Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-discourse (authorship Pub dates, etc.) Topics Other chat
  • 25. ? • ISEQ - psychometrics, etc. • Outcome data (IRR) • Trust • Predictive value of summary data (e.g. pages->Topics) • Sequences • Deep dive – key patterns, proof of concept of those patterns. Critical incidents.
  • 26. Thank you Simon Knight http://sjgknight.com @sjgknight Knight, S., Littleton, K., (2015). Developing a multiple- document-processing performance assessment for epistemic literacy. In The 5th International Learning Analytics & Knowledge Conference (LAK15): Scaling Up: Big Data to Big Impact, Poughkeepsie, NY. http://oro.open.ac.uk/41711/
  • 27. 27 Predictors? GPA, topic knowledge, search skill Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?) Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-data (authorship Pub dates, etc.) Topics Sources (http://) Other chat
  • 28. @sjgknight www.sjgknight.com Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search, Presented at the 2nd Society of the Query Conference, Amsterdam • Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K., (2014). Epistemic Networks for Epistemic Commitments. Presented at the International Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39254/ • Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39226 (HEA presentation version Blog/presentation) • Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39181/ • Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R. Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research on Teaching Thinking. Routledge. • In draft publications on computational approaches to discourse analysis + social account of epistemic cognition
  • 29. References Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007 Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5 Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media Institute: The Open University, UK. Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources: Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027 Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Information Age Publishing (IAP). Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning. Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012 Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior Research Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0 Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centered design. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17). Rowman & Littlefield. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001 Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information.Metacognition and Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2 OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdf Rouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge. Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039 Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003 Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information? Retrieved from http://dspace.ou.nl/handle/1820/4390 Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030
  • 30. Task1 (warmup) Please type the answers to the following three prompts in your Task Pad (click in the bar at the top of the browser). You may use the internet to find the answers. • In 2010 what was the educational expenditure per primary student in [XXX] as a % of GDP? • In 2010 what was the total health expenditure as a % of GDP in [XXX]? • How much (in US dollars) does a big mac cost in [ZZZ]? Warmup – variables, – XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana, Guatemala, El Salvador, Ecuador (list of countries not too far from top of alphabet, with data for 2010) – Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada, Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index) If you find the warmup taking too long (over 10 minutes) but you feel you’re now comfortable with using Coagmento, you should move on to Task 1. • Task Submission Procedure [Slide 9]
  • 31. MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate. Your task is to act as an advisor to an official within the science ministry. You are advising an official on the issues below. The official is not an expert in the area, but you can assume they are a generally informed reader. They are interested in the best supported claims in the documents. Produce a summary of the best supported claims you find and explain why you think they are. Note you are not being asked to “create your own argument” or “summarise everything you find” but rather, make a judgement about which claims have the strongest support. A colleague has already found a number of documents for you to process with your partner, you should use these to extract the best supported claims (without using the internet to find further material). You should: Read the questions/topic areas provided, these will require you to find information and arguments in the documents to present the best supported of these, you should decide with your partner which are best as you read. Group information together by using headings in the Editor You should work with your partner to explain why the claims you’ve found are the best available You should spend about 45 minutes on this task A review is coming up for the license of Glyphosate, the official would like to know the best supported claims around its risks. A colleague has collected some documents, available from the

Editor's Notes

  1. http://www.dailymail.co.uk/health/article-1025277/Quack-medicine-Peking-duck-better-heart-statins.html http://www.theguardian.com/science/controversiesinscience http://www.theguardian.com/environment/blog/2013/jun/17/wait-conclusive-science-concerned-issue
  2. the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012). Not ‘digital’ in sense of coding, etc. Interest is in rich understanding of literacy Internet use prevalent, as are difficulties in information seeking using the internet
  3. http://www.dailymail.co.uk/health/article-1025277/Quack-medicine-Peking-duck-better-heart-statins.html http://www.theguardian.com/science/controversiesinscience http://www.theguardian.com/environment/blog/2013/jun/17/wait-conclusive-science-concerned-issue http://www.nhs.uk/news/2008/06June/Pages/Redyeastriceandheartdisease.aspx
  4. http://www.coagmento.org/index.php
  5. the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012).
  6. They argue that we need two foci: Productive use of metadata and meta-discourse – credentials, dates, source locations, quote v paraphrase, citations, primary/secondary source, etc. are all important parts of the discourse, and the discourse around this becomes an object for discourse (meta-discourse) too Use of authoritative sources (i.e. stating claims, and citing sources), with a focus on discourse for idea improvement and knowledge-creation
  7. https://addons.mozilla.org/ga-ie/firefox/addon/coagmento/
  8. So, in my context, we might imagine a student saying “we should try looking on W for that” – so we know something about how they’re thinking about seeking/search already
  9. We can see their query (and where it’s conducted)
  10. Images public domain
  11. Images public domain
  12. Participants given 2 texts of varying quality are more likely to favor high quality sources, BUT they make little reference to source features (on average only 1.85 out of 10 features); and rarely (<6% of the 154 participants) explicitly use source information for justifying their evaluation of the text’s explanation Kobayashi [26] indicated through a controlled experimental design, while:
  13. the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012).
  14. And particularly interested in this in the context of information seeking, but rejecting the ‘beliefs’ element, instead focussing on the commitments to epistemic standards implicit (or explicit) in student’s seeking processes Lower image from http://commons.wikimedia.org/wiki/File:LensFilter-001.jpg (cc-by) http://commons.wikimedia.org/wiki/File:Dichroic_filters.jpg public domain (by NASA)
  15. How simple or complex knowledge is – can it be compartamentalised, or is it interconnected? The justification for knowledge – does knowledge come from observation, or from rules of enquiry? The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others? The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct. https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen "Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique https://openclipart.org/detail/62989/scales-of-justice-by-laobc https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom https://openclipart.org/detail/203314/chemical-by-Minduka https://openclipart.org/detail/202707/1412278291-by-pnx https://openclipart.org/detail/202678/1412242970
  16. How simple or complex knowledge is – can it be compartamentalised, or is it interconnected? The justification for knowledge – does knowledge come from observation, or from rules of enquiry? The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others? The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct. https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen "Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique https://openclipart.org/detail/62989/scales-of-justice-by-laobc https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom https://openclipart.org/detail/203314/chemical-by-Minduka https://openclipart.org/detail/202707/1412278291-by-pnx https://openclipart.org/detail/202678/1412242970 http://pixabay.com/en/search-box-optic-http-www-13476/
  17. break point analysis -e.g. shifts in language re: Britte Source v I think - look for sequences of source->i think, i think->source What behavioural patterns would we hypothesise? E.g. page->copy->etherpad (Synth-list) Chat co-occurrence data ( Query->page->query->page (SDiv/topic)   SDiv Topic SQual Synth page-vis CIS   CIS MDP   page-use MDP (Hyp) MDP     Chat-Explor CIS MDP MDP CIS MDP (Hyp)   Chat-sourQual (Hyp)   MDP (Hyp) MDP Chat-Topic CIS MDP CIS MDP (Hyp) CIS MDP CIS TextLen CIS MDP CIS MDP CIS  
  18. How simple or complex knowledge is – can it be compartamentalised, or is it interconnected? The justification for knowledge – does knowledge come from observation, or from rules of enquiry? The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others? The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct. https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen "Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique https://openclipart.org/detail/62989/scales-of-justice-by-laobc https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom https://openclipart.org/detail/203314/chemical-by-Minduka https://openclipart.org/detail/202707/1412278291-by-pnx https://openclipart.org/detail/202678/1412242970 http://pixabay.com/en/search-box-optic-http-www-13476/
  19. NOTE TO US: For 1, building off http://searchresearch1.blogspot.co.uk/2014/05/wednesday-search-challenge-52014-how.html, best way to answer ‘2’ is to just use the same website. For 3 expect to see ‘big mac index’ or/and www.numbeo.com http://www.statista.com/statistics/274326/big-mac-index-global-prices-for-a-big-mac/