Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evidence-Centered Approach to Online Assessment of Students’ Digital Competence


Published on

Presentation at TEA 2016 conference, Tallinn University

Published in: Education
  • Be the first to comment

  • Be the first to like this

Evidence-Centered Approach to Online Assessment of Students’ Digital Competence

  1. 1. Evidence-Centered Approach to Online Assessment of Students’ Digital Competence Mart Laanpere, Kai Pata (Tallinn University) Mario Mäeots , Leo Siiman (University of Tartu)
  2. 2. Problem space • Need to assess digital competence: a newly defined key competences in Estonian national curriculum for primary and secondary schools • Requirements: – Compliant with DigComp framework – Implemented online using e-exam system EIS – Scalability – Automated assessment – Reliability, validity
  3. 3. Digital competence • Three competing approaches to digital competence/competencies: – Theory-based knowledge, derived from the academic discipline of computer science (computational thinking) – Pragmatic skills, needed at the workplace (ECDL) – Key competence, contextualised, closer to “learning to learn” skill • DigComp follows the second approach, Estonian curriculum combines 2nd and 3rd
  4. 4. Teaching and assessing ICT skills in Estonia • 1986: Computer science as a compulsory theoretical subject, no exam, teacher-created tests • 1996: Informatics as an elective subject, learning generic ICT skills for office jobs • 2000: ICT and media as cross-curricular theme, pilot exam on national level (test + practical task) • 2011: informatics as an elective subject, digital competence as a key competence
  5. 5. Research design and organisation • Two teams of researchers from two different universities (University of Tartu & Tallinn University) • Combination of two approaches: – Inductive/descriptive, bottom-up (Tartu team): collecting and analysing the best practice from large-scale comparative and national studies – Deductive/prescriptive, top-own (Tallinn team): building a new framework from scratch, deducing the new type of practice
  6. 6. ECD framework (Almond, Mislevy et al. 2015) inspired by e-exams of Cisco Network Academy; Has been applied in various contexts: Assessing language competence, Inquiry skills in STEM Etc.
  7. 7. Layered assessment architecture of ECD Layer Role Key entities Examples Domain analysis Collect info about domain, teaching practice, frames Concepts; terminology; tools; representation forms Curricula, rubrics, test Domain modelling Express assessment argument in narrative form KSAs; potential work products; potential observations Design patterns Assessment framework CAF Express assessment argument as blueprints for tasks or items Student, evidence, and task models; student- model, rubrics etc Task templates Imple- mentation Implement assessment, presenting tasks or items and gathering and analyzing responses Task materials (incl.all materials, tools, affordances); work products Rendering protocols for tasks; IMS/QTI representation of materials & scores; Delivery Coordinate interactions of students and tasks; task and test-level scoring; reporting. Tasks as presented; work products as created; scores as evaluated. summaries for individual and group- level reports;
  8. 8. Assessment framework (CAF) What? How? Where? Extent?
  9. 9. Domain analysis: example
  10. 10. Performance taxonomy Fact Concept Procedure Rule Knows Understands Applies HOTS Based on: Merrill ja Feisel-Schmitz
  11. 11. Bayesian network Information & Data Literacy Digital content creation Communication & collaboration Digital safety Problem solving
  12. 12. KSA (Knowledge-Skills-Abilities) P_Info P_Comm P_Content P_Safety P_Probl T_Info T_Comm T_Content T_Safety T_Probl
  13. 13. Example: practical task • Create a presentation that compares two countries: Estonia and Slovenia • Criteria: – At least 4 slides/screens – Diagrams based on reliable and recent data – Images (IPR need to be followed, licensing)
  14. 14. Piloting • Not completed, planned in 6-8 schools this October • Bayesian data analysis, evidence accumulation • Scenario-based design of new task templates • Exploring the solutions for simulation tasks (inspired by examples collected by Tartu group)
  15. 15. Questions for discussion • Any experiences with ECD? • How to combine inductive and deductive approaches? • Semi-automated assessment of practical tasks?