Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Developing a multiple-document-processing performance assessment for epistemic literacy

The LAK15 theme “shifts the focus from data to impact”, noting the potential for Learning Analytics based on existing technologies to have scalable impact on learning for people of all ages. For such demand and potential in scalability to be met the challenges of addressing higher-order thinking skills should be addressed. This paper discuses one such approach – the creation of an analytic and task model to probe epistemic cognition in complex literacy tasks. The research uses existing technologies in novel ways to build a conceptually grounded model of trace-indicators for epistemic-commitments in information seeking behaviors. We argue that such an evidence centered approach is fundamental to realizing the potential of analytics, which should maintain a strong association with learning theory.

  • Login to see the comments

  • Be the first to like this

Developing a multiple-document-processing performance assessment for epistemic literacy

  1. 1. Developing a multiple-document- processing performance assessment for epistemic literacy Simon Knight Karen Littleton @sjgknight
  2. 2. Developing a multiple-document- processing performance assessment for epistemic literacy Simon Knight Karen Littleton Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah, Simon Buckingham Shum, Bart Rienties, Fridolin Wild @sjgknight
  3. 3. Introduction • New technologies: – Foreground issues in literacy – finding & comprehending multi- media – Afford new opportunity to explore & tackle these issues
  4. 4. What is [Digital] literacy? • [Digital] literacy – – Comprehension in context of selecting & processing rich online multimedia content with diverse sources (OECD, 2013)
  5. 5. Tasks • Two collaborative tasks facilitated by a browser addon • One group provided with documents; the second group searches on the web • “A review of the best supported claims around the risks” of a substance (herbicide or food supplement)
  6. 6. 7 Friends of the Earth: Press Release (Urine presence) FoE Commission ed report (‘scientific’) (-ve) Science- Literacy website: Refutation (+ve) Farmer’s Weekly Reprints (+ve) Related peer-review publication (Limited risk) Peer-review publication Health danger Reuters Reprints main claims Blogger Critiques journal & author Peer-review publication (Limited health risks) Peer-review review of literature (Limited risk to health or plants) Peer-review of lit (Limited risk; control suggestions) Urine Health Agricu lture
  7. 7. 8 Statins Citrinin contamination Concentration
  8. 8. Skills of literacy Rouet [39] – students should be taught: • Skill of integration: the ability to connect prior and new information, including across documents, and including where claims are inconsistent or contradictory • Skill of sourcing: the ability to identify parameters that characterize the author and conditions of production of the information • Skill of corroboration: the ability to check information against multiple sources for its accuracy
  9. 9. Skills of literacy • Communication – particularly meta-discourse involving exploratory discussion of credentials, sources, etc. – is key to collectively authored written outputs (Goldman and Scardamalia [17, p.260]).
  10. 10. Coagmento Tool • Chat • Foreground searches • Share ‘snippets’ • Etherpad • Tracks pageviews & copy (/ctrl+c)
  11. 11. Epistemic Commitments we should try looking on Wikipedia for that
  12. 12. 13
  13. 13. A C B ¬A B ……… ……… … 14
  14. 14. A C By x (2002) ……… ……… … B ¬A By y (2014) B By Gov (nd) 15
  15. 15. Source features • Participants given 2 texts of varying quality are more likely to favor high quality sources, • BUT they make little reference to source features (on average only 1.85 out of 10 features); • and rarely (<6% of the 154 participants) explicitly use source information for justifying their evaluation of the text’s explanation Kobayashi [26](2014)
  16. 16. Epistemic commitments • Multiple Documents—Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet/Rouet & Britt) • 5 Steps: 1. Model construction 2. information need assessment; 3. document processing; - relevance, update needs, seek new information 4. task product creation; 5. task product assessment.
  17. 17. “exploring students’ thought processes during online searching allows examination of personal epistemology not as a decontextualized set of beliefs, but as an activated, situated aspect of cognition that influences the knowledge construction process” (Hofer, 2004, p. 43). The Lens of Epistemic Beliefs Commitments
  18. 18. Certainty 19 A C By x (2002) …… …… …… … Source A B C A B C Simplicity Justification 1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected? 2.The justification for knowledge – does knowledge come from observation, or from rules of enquiry? 3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others? 4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute
  19. 19. Known variables (GPA, etc.) • ISEQ • GPA • Self-report prior knowledge • Self-report search skill • Self-report partner familiarity
  20. 20. Trace constructs • n Pages visited • n Pages used • Page-dwell >= 30 seconds • n Pages visited/N pages visited Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-discourse (authorship Pub dates, etc.) Topics Other chat • n Pages used/N pages used • Query distances • Coded-chat instances (topic-terms, source-terms, exploratory-terms) • Collaborative-symmetry on <<&^^^
  21. 21. Outcome constructs • 1-3 score on: – Topic coverage – Source diversity (largely ‘3’) – Source quality/evaluation – Synthesis • 1-12 total score 22 Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?)
  22. 22. ProjectID ISEQPaired GPAPair SearchSkillPair PriorKnowlPair PartnerFamilPair SynthScore TopicScore SourDivScore SourQualScore TotatlScore n-pages visited n-pages used n-dwelled Chat-n-messages Chat-n-explor Chat-n-topic Chat-n-source queryDists (CIS ONLY) Partner Symmetry Outcome vars Process vars Control vars
  23. 23. 24 Predictors? GPA, topic knowledge, search skill Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?) Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-discourse (authorship Pub dates, etc.) Topics Other chat
  24. 24. ? • ISEQ - psychometrics, etc. • Outcome data (IRR) • Trust • Predictive value of summary data (e.g. pages->Topics) • Sequences • Deep dive – key patterns, proof of concept of those patterns. Critical incidents.
  25. 25. Thank you Simon Knight @sjgknight Knight, S., Littleton, K., (2015). Developing a multiple- document-processing performance assessment for epistemic literacy. In The 5th International Learning Analytics & Knowledge Conference (LAK15): Scaling Up: Big Data to Big Impact, Poughkeepsie, NY.
  26. 26. 27 Predictors? GPA, topic knowledge, search skill Source Diversity Source Quality/ Evaluation Synthesis Topic coverageOutcome: (Peer assess?) Pages visited Pages used Collaborative symmetry Query diversity Chat data Exploratory/ epistemic Meta-data (authorship Pub dates, etc.) Topics Sources (http://) Other chat
  27. 27. @sjgknight Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search, Presented at the 2nd Society of the Query Conference, Amsterdam • Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K., (2014). Epistemic Networks for Epistemic Commitments. Presented at the International Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open University Eprint Archive: • Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Available via The Open University Eprint Archive: (HEA presentation version Blog/presentation) • Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education. Available via The Open University Eprint Archive: • Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R. Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research on Teaching Thinking. Routledge. • In draft publications on computational approaches to discourse analysis + social account of epistemic cognition
  28. 28. References Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007 Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5 Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media Institute: The Open University, UK. Retrieved from Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources: Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027 Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Information Age Publishing (IAP). Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning. Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012 Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior Research Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0 Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centered design. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17). Rowman & Littlefield. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001 Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information.Metacognition and Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2 OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from Rouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge. Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039 Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003 Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information? Retrieved from Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030
  29. 29. Task1 (warmup) Please type the answers to the following three prompts in your Task Pad (click in the bar at the top of the browser). You may use the internet to find the answers. • In 2010 what was the educational expenditure per primary student in [XXX] as a % of GDP? • In 2010 what was the total health expenditure as a % of GDP in [XXX]? • How much (in US dollars) does a big mac cost in [ZZZ]? Warmup – variables, – XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana, Guatemala, El Salvador, Ecuador (list of countries not too far from top of alphabet, with data for 2010) – Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada, Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index) If you find the warmup taking too long (over 10 minutes) but you feel you’re now comfortable with using Coagmento, you should move on to Task 1. • Task Submission Procedure [Slide 9]
  30. 30. MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate. Your task is to act as an advisor to an official within the science ministry. You are advising an official on the issues below. The official is not an expert in the area, but you can assume they are a generally informed reader. They are interested in the best supported claims in the documents. Produce a summary of the best supported claims you find and explain why you think they are. Note you are not being asked to “create your own argument” or “summarise everything you find” but rather, make a judgement about which claims have the strongest support. A colleague has already found a number of documents for you to process with your partner, you should use these to extract the best supported claims (without using the internet to find further material). You should: Read the questions/topic areas provided, these will require you to find information and arguments in the documents to present the best supported of these, you should decide with your partner which are best as you read. Group information together by using headings in the Editor You should work with your partner to explain why the claims you’ve found are the best available You should spend about 45 minutes on this task A review is coming up for the license of Glyphosate, the official would like to know the best supported claims around its risks. A colleague has collected some documents, available from the