Web-based Self- and Peer Assessment  of Teachers Digital Competences          Mart Laanpere, Hans Põldoja
cbaThis work is licensed under the Creative CommonsAttribution-ShareAlike 3.0 Unported License. To view a copyof this lice...
Research context•   Importance of educational technology    competencies•   Generic ICT competency frameworks (e.g. ICDL) ...
Research problemTo what extent and how could beteachers’ educational technologycompetencies assessed using a Web-based tool?
Existing competency            frameworks•   International Computer Driving License•   UNESCO ICT Competency Framework for...
• ECDL / ICDL• Used in 148  countries• Focused on ICT  usage• Standardized  testing
• Launched 2008,  revised 2011• Guidelines for  creating national  competency models• 6 subdomains
• ISTE NETS-T 2008• 20 competencies in  5 competency  groups• Used in Norway,  Netherlands,  Finland, etc.
Design challenges
Design challenges•   How to define performance indicators of all    competencies in ETCM?•   How to select appropriate meth...
Educational Technology Competency   Model for Estonian Teachers•   Based on ISTE NETS-T 2008•   5-level assessment rubric ...
Measuring Educational    Technology Competencies•   Assessment methodology and instruments must be    reliable, valid, flex...
Web-based Assessment of        Competencies•   Five-dimensional framework for authentic assessment (Gulikers et    al, 200...
Design anddevelopment of   DigiMina
http://trac.htk.tlu.ee/digimina/
Research-based design methodologyContextual Inquiry   Participatory Design     Product Design           Software Prototype...
Personas•   Teacher training master student•   Novice teacher•   Experienced teacher•   Educational technologist of a scho...
Scenarios•   Master student is evaluating her educational    technology competencies•   Peer assessment of problem solving...
Stsenaarium 2: Pädevuste partnerhindamineKaisa on noor matemaatika õpetaja, kes on esimest aastat koolis. Õpetaja kutseaas...
Participatory design sessions                   •   2 sessions                   •   Discussing the                       ...
Main concepts
Competency Test•   Competency test can be taken several times to    measure the advancement•   Usability issue: large numb...
Tasks•   Task types:    •   automatically assessed self-test items    •   peer assessment task    •   self reflection task•...
Competency Profile•   Level of competencies is displayed as a diagram•   User can compare her competency level with the    ...
Group•   Typically created for a school or a group of    teacher training students•   Group owner can see competency profil...
Competency Requirements•   Large number of competency profiles would make    DigiMina a valuable planning tool•   Competenc...
Current prototype
Task development
Assessment rubrics example3.1. Demonstrate fluency in technology systems and the transfer of current knowledge to newtechno...
Example self-test task•   Screen recording of a teacher joining national    educational portal Koolielu and making several...
Example peer review task•   Teacher must adapt a given study guide to her own    working context (age range, subject area,...
Example self-reflection task•   4.1 level 1 — participates as a partner in a    multicultural project
Development of test items•   Test items are authored using IMS QTI compatible    test authoring tool TATS (Tomberg & Laanp...
Future work
Development of DigiMina•   Visualizations and analytics•   Support for additional QTI question types:    orderInteraction ...
References• Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A Five-Dimensional Framework for  Authentic A...
Thank you!•   mart.laanpere@tlu.ee•   hans.poldoja@tlu.ee•   http://trac.htk.tlu.ee/digimina/
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
Upcoming SlideShare
Loading in …5
×

Web-based Self- and Peer Assessment of Teachers Digital Competences

1,396 views

Published on

Presentation in the research group seminar, Institute of Informatics, Tallinn University, 7 March 2012.

Based on the following publication:

Põldoja, H., Väljataga, T., Tammets, K., & Laanpere, M. (2011). Web-based Self- and Peer- assessment of Teachers’ Educational Technology Competencies. In H. Leung, E. Popescu, Y. Cao, R. Lau, & W. Nejdl (Eds.), Advances in Web-Based Learning – ICWL 2011: 10th International Conference, Hong Kong, China, December 2011. Proceedings (pp. 122–131). Berlin / Heidelberg: Springer. http://www.springerlink.com/content/e3t2042568271213/

Published in: Education, Technology

Web-based Self- and Peer Assessment of Teachers Digital Competences

  1. 1. Web-based Self- and Peer Assessment of Teachers Digital Competences Mart Laanpere, Hans Põldoja
  2. 2. cbaThis work is licensed under the Creative CommonsAttribution-ShareAlike 3.0 Unported License. To view a copyof this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 444 CastroStreet, Suite 900, Mountain View, California, 94041, USA.
  3. 3. Research context• Importance of educational technology competencies• Generic ICT competency frameworks (e.g. ICDL) do not cover all the competencies needed for educational use of ICT• Educational Technology Competency Model (ETCM) for Estonian teachers• DigiMina project for assessing teachers’ educational technology competencies
  4. 4. Research problemTo what extent and how could beteachers’ educational technologycompetencies assessed using a Web-based tool?
  5. 5. Existing competency frameworks• International Computer Driving License• UNESCO ICT Competency Framework for Teachers• ISTE National Educational Technology Standards for Teachers (NETS-T)
  6. 6. • ECDL / ICDL• Used in 148 countries• Focused on ICT usage• Standardized testing
  7. 7. • Launched 2008, revised 2011• Guidelines for creating national competency models• 6 subdomains
  8. 8. • ISTE NETS-T 2008• 20 competencies in 5 competency groups• Used in Norway, Netherlands, Finland, etc.
  9. 9. Design challenges
  10. 10. Design challenges• How to define performance indicators of all competencies in ETCM?• How to select appropriate methods and instruments for assessing competencies?• How to implement selected assessment methods in a Web-based tool?
  11. 11. Educational Technology Competency Model for Estonian Teachers• Based on ISTE NETS-T 2008• 5-level assessment rubric developed by local expert group
  12. 12. Measuring Educational Technology Competencies• Assessment methodology and instruments must be reliable, valid, flexible, but also affordable with respect to time and costs.• Four levels of measuring competencies (Miller, 1990): • knows • knows how • shows how • does
  13. 13. Web-based Assessment of Competencies• Five-dimensional framework for authentic assessment (Gulikers et al, 2004): • tasks: meaningful, relevant, typical, complex, ownership of problem and solution space; • physical context: similar to professional work space and time frame, professional tools; • social context: similar to social context of professional practice (incl. decision making); • form: demonstration and presentation of professionally relevant results, multiple indicators; • criteria: used in professional practice, related to realistic process/ product, explicit
  14. 14. Design anddevelopment of DigiMina
  15. 15. http://trac.htk.tlu.ee/digimina/
  16. 16. Research-based design methodologyContextual Inquiry Participatory Design Product Design Software Prototype As Hypothesis Concept mapping User stories Information architecture Participatory design sessions Paper prototyping Scenario-based High-fidelity Personas Agile sprints design prototyping Adapted from (Leinonen et al, 2008)
  17. 17. Personas• Teacher training master student• Novice teacher• Experienced teacher• Educational technologist of a school• Trainings manager (in a national organization) (Cooper et al, 2007)
  18. 18. Scenarios• Master student is evaluating her educational technology competencies• Peer assessment of problem solving tasks• Educational technologist of a school is getting an overview of teachers’ educational technology competencies• Training manager is compiling a training group with sufficient level of competencies (Carroll, 2000)
  19. 19. Stsenaarium 2: Pädevuste partnerhindamineKaisa on noor matemaatika õpetaja, kes on esimest aastat koolis. Õpetaja kutseaastakäigus tuleb muuhulgas tähelepanu pöörata ka oma pädevuste kaardistamisele. Kaisa onkorra varem ülikooli ajal juba oma haridustehnoloogilisi pädevusi hinnanud, kuidnüüd koolis on ta oma teadmisi ja oskusi praktikas saanud rakendada.DigiMina keskkonda sisenedes näeb ta oma eelmisel aastal tehtud pädevuste hindamisetesti. Ta saab valida, kas alustab pädevuste testi täitmist eelmise aasta tasemeltvõi päris algusest. Talle meenub, et eelmisel korral võttis kõigi ülesannetelahendamine suhteliselt palju aega ning ta otsustab sel korral alustada testitäitmist eelmise aasta tasemelt. Sel korral on lisaks valikvastustega küsimustele kaprobleemülesandeid, mille puhul tuleb vabas vormis omapoolne lahendus kirjutada.Vastamise lõpus kuvatakse pädevusprofiili diagramm, kuid osad pädevused on sealtpuudu. Kaisa loeb kõrvalolevalt märkuselt, et vabas vormis küsimusi peavad hindamateised DigiMina kasutajad ning täielikke tulemusi kuvatakse alles siis kui temavastused on hinnatud.Uuesti avalehele jõudes kuvatakse Kaisale teadet, et ta võib osaleda teistekasutajate poolt sisestatud probleemülesannete lahenduste hindamises. Kaisale pakubhuvi, kuidas teised etteantud probleeme lahendasid, ning ta otsustab ühtepädevustesti hinnata. Ülesande hindamisel peab Kaisa lisama omapoolse kommentaaripakutud lahendusele ning valima menüüst hindepunktid. Hindamise lihtsustamisekskuvatakse talle ka hindamiskriteeriume. Kui esimene vastus on hinnatud otsustab tahinnata veel paari vastust. Kaisa ei näe, kelle vastuseid ta hindas, kuid ta oletab,et tegemist võis olla teiste õpetaja kutseaasta üliõpilastega.Paari päeva pärast saab Kaisa e-maili, et tema vastused on kahe DigiMina kasutajapoolt hinnatud. Sisselogides näeb ta oma vastuste kohta kirjutatud lühikesikommentaare ning pääseb kokkuvõtva diagrammi juurde. Kokkuvõttes saab ta võrreldaoma pädevuste taset ka õpetaja kutseaasta üliõpilaste keskmise pädevustasemega ningkõigi kasutajate keskmise pädevustasemega.
  20. 20. Participatory design sessions • 2 sessions • Discussing the scenarios • Drawing the sketches
  21. 21. Main concepts
  22. 22. Competency Test• Competency test can be taken several times to measure the advancement• Usability issue: large number of tasks (20 competencies, 5 levels)• Solutions: • Can be saved and continued later • Setting the starting level with self-evaluation
  23. 23. Tasks• Task types: • automatically assessed self-test items • peer assessment task • self reflection task• Need to increase the number of competencies that can be assessed with a self-test• Peer assessment requires blind review from a user in a same or higher competency level
  24. 24. Competency Profile• Level of competencies is displayed as a diagram• User can compare her competency level with the average level of various groups• Privacy settings (private, group members, public)• Can be linked or embedded to external web pages
  25. 25. Group• Typically created for a school or a group of teacher training students• Group owner can see competency profiles of all members• Anybody can create a group• Groups can be set up as private or public
  26. 26. Competency Requirements• Large number of competency profiles would make DigiMina a valuable planning tool• Competency requirements can be created by the training manager, teacher trainer and group owner• Will be implemented in a later phase
  27. 27. Current prototype
  28. 28. Task development
  29. 29. Assessment rubrics example3.1. Demonstrate fluency in technology systems and the transfer of current knowledge to newtechnologies and situations 1 2 3 4 5Creates a user Manages access Solves Transfers working Choosesaccount in a web- rights to the independently the methods from (compares,based system and resources problems that known web evaluates) the mostcreates/uploads published in the occur during the environment/ suitable tool for aresources; uses web. use of ICT tools software to an given task.common software/ (using help, manual, unknownweb environments/ FAQ or forums environment.hardware with the when needed);help of a user combines differentmanual; uses tools; changes thepresentation tools settings of a web-and a printer; based system.saves/copies files toexternal drive.
  30. 30. Example self-test task• Screen recording of a teacher joining national educational portal Koolielu and making several errors during the process• Multiple response question about the errors made
  31. 31. Example peer review task• Teacher must adapt a given study guide to her own working context (age range, subject area, software)
  32. 32. Example self-reflection task• 4.1 level 1 — participates as a partner in a multicultural project
  33. 33. Development of test items• Test items are authored using IMS QTI compatible test authoring tool TATS (Tomberg & Laanpere, 2011)• 3 IMS QTI question types currently supported: • choiceInteraction (multi-choice) • choiceInteraction (multi-response) • extendedTextInteraction
  34. 34. Future work
  35. 35. Development of DigiMina• Visualizations and analytics• Support for additional QTI question types: orderInteraction and associateInteraction• Competency requirements• Integration with Koolielu portal
  36. 36. References• Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A Five-Dimensional Framework for Authentic Assessment. Educational Technology Research & Development, 52(3), 67–86.• Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63–S67.• Leinonen, T., Toikkanen, T., & Silvfast, K. (2008). Software as Hypothesis: Research-Based Design Methodology. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (pp. 61–70). Indianapolis, IN: Indiana University.• Cooper, A., Reimann, R., & Cronin, D. (2007). About Face 3: The Essentials of Interaction Design. Indianapolis, IN: Wiley Publishing, Inc.• Carroll, J. M. (2000). Making Use: Scenario-Based Design of Human-Computer Interactions. Cambridge, MA: The MIT Press.• Tomberg,V., & Laanpere, M. (2011). Implementing Distributed Architecture of Online Assessment Tools Based on IMS QTI ver.2. In F. Lazarinis, S. Green, & E. Pearson (Eds.), Handbook of Research on E-Learning Standards and Interoperability: Frameworks and Issues (pp. 41–58). IGI Global.
  37. 37. Thank you!• mart.laanpere@tlu.ee• hans.poldoja@tlu.ee• http://trac.htk.tlu.ee/digimina/

×