Novice e-ass

644 views
543 views

Published on

NOVICE summer school 2011
(www.noviceproject.eu)
Workshop: E-assessment

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
644
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Novice e-ass

  1. 1. Jan Ehlers<br />E-Assessment<br />
  2. 2. Jan Ehlers<br />Workshop: E-Assessment<br /><ul><li>Whatis e-assessment?
  3. 3. Someexperiences
  4. 4. Whatcouldbedoneelectronically?
  5. 5. The ideal exam</li></li></ul><li>Introduction<br />Who areyou?<br />Name, location, interests in assessment<br />26.08.2011<br />3<br />
  6. 6. Assessment-formats<br />26.08.2011<br />4<br /><ul><li>diagnostic
  7. 7. formative
  8. 8. summative</li></li></ul><li>Group work - brainstorming<br />Whatkindofassessments do you do atyouruniversity?<br /><ul><li>Threegroups
  9. 9. Diagnostic/formative/summative
  10. 10. Changingflipchartsevery 3-5 minutes</li></ul>26.08.2011<br />5<br />
  11. 11. 26.08.2011<br />6<br />RUEDEL C (2009): Was ist eAssessment? Hamburger eLMAGAZIN (02), 22-24 <br />
  12. 12. Traditional vs. electronical<br />These examsarequitethe same:<br />Traditional Electronical<br />Minutes / term papers ----------<br />Written tests (open or closed questions) ----<br />Oral exams ------------------------<br />Portfolios ---------------------------<br />Peer-assessment ----------------<br />Scientific practicals --------------<br />Study diary ------------------------<br />26.08.2011<br />7<br />Electronic submissions<br />E-assessment, e-exam<br />Adaptive testing<br />E-portfolios<br />Peer-assessment<br />Simulations<br />Weblog<br />
  13. 13. Traditional vs. electronical<br />These examsarequitesimilar:<br />Traditional Electronical<br />Oral collaboration ----------------<br />Group examination --------------<br />Presentations----------------------<br />26.08.2011<br />8<br />Discussionboards/voting<br />Wikis<br />Digital storytelling<br />
  14. 14. Diagnosticassessment<br />Selectionprocedureoftheuniversity<br />26.08.2011<br />9<br /><ul><li>since 2006/07 psychologicalmotivation-test
  15. 15. 883 firstyearstudentsfrom  6,553 applicants
  16. 16. 180 items/30 min
  17. 17. Likert-values 1-6</li></li></ul><li>Formative assessment<br />CASUS und PowerVote<br />26.08.2011<br />10<br />voluntary<br />usedby ~75 %<br />BÖRCHERS M, TIPOLD A, PFARRER C, FISCHER MR, EHLERS JP: Akzeptanz von fallbasiertem, interaktiven eLearning in der Tiermedizin am Beispiel des CASUS-Systems. Tierärztliche Praxis K 38, 6 (2010) 379-388<br />EHLERS JP, MÖBS D, VOR DEM ESCHE J, BLUME K, BOLLWEIN H, TIPOLD A:Einsatz von formativen, elektronischen Testsystemen in der Präsenzlehre.  GMS Z MedAusbild. 2010, 27 (4), Doc 59<br />
  18. 18. What are your exams like?<br />0<br />1 - Oral exams<br />2 - Practicalexams<br />3 - Writtenexams<br />4 - Electronic Exams<br />5 - Essay questions<br />6 - Multiple choice<br />7 - Others<br />
  19. 19. What are your exams like?<br />26.08.2011<br />12<br />0%<br />1 - Oral exams<br />0%<br />2 - Practicalexams<br />0%<br />3 - Writtenexams<br />0%<br />4 - Electronic Exams<br />5 - Essay questions<br />0%<br />6 - Multiple choice<br />0%<br />7 - Others<br />0%<br />
  20. 20. You want your students to document their EMS and present these documentations afterwards to others. Which tool could help you here?<br />26.08.2011<br />13<br />0<br />1 - Online surveys<br />2 - Wikis<br />3 - Discussionboards<br />4 - Blogs<br />5 - Portfolios<br />6 - SocialBookmarking<br />7 - Facebook<br />
  21. 21. You want your students to document their EMS and present these documentations afterwards to others. Which tool could help you here?<br />26.08.2011<br />14<br />1 - Online surveys<br />0%<br />2 - Wikis<br />0%<br />0%<br />3 - Discussionboards<br />4 - Blogs<br />0%<br />5 - Portfolios<br />0%<br />6 - SocialBookmarking<br />0%<br />0%<br />7 - Facebook<br />26.08.2011<br />14<br />
  22. 22. Summative assessment<br />Q [kju:] (Codiplan)<br />26.08.2011<br />15<br />Start: August 2008<br />126 exams<br />25,443 individual exams<br />9,602 questions<br />Partlyblendedassessment<br />EHLERS JP, CARL T, WINDT K-H, MÖBS D, REHAGE J, TIPOLD A: BlendedAssessment: Mündliche und elektronische Prüfungen im klinischen Kontext. In: Zeitschrift für Hochschulentwicklung 4, 3 (2009) 24-36<br />
  23. 23. Requirementsfor summative assessments<br />Quality criteria<br />26.08.2011<br />16<br />Utility<br />Testfairness<br />Testeconomy<br />Transparency<br />Reasonabilty<br />Objectivity<br />Reliability<br />Validity<br />
  24. 24. What is for you most important?<br />26.08.2011<br />17<br />00<br />1 - Objectivity<br />2 - Reliability<br />3 - Validity<br />4 - Utility<br />5 - Testfairness<br />6 - Testeconomy<br />7 - Transparency<br />8 - Reasonability<br />
  25. 25. What is for you most important?<br />26.08.2011<br />18<br />1 - Objectivity<br />0%<br />2 - Reliability<br />0%<br />3 - Validity<br />0%<br />4 - Utility<br />0%<br />5 - Testfairness<br />0%<br />6 - Testeconomy<br />0%<br />7 - Transparency<br />0%<br />8 - Reasonability<br />0%<br />26.08.2011<br />18<br />
  26. 26. Group work - brainstorming<br />Whataretheadvantages, disadvantagesandmaybe „visions“ of electronic exams?<br /><ul><li>Threegroups
  27. 27. Advatages/disadvantages/“visions“
  28. 28. Changingflipchartsevery 3-5 minutes</li></ul>26.08.2011<br />19<br />
  29. 29. Students´ view<br />
  30. 30. Acceptance summative e-assessment<br />26.08.2011<br />21<br />Was the system easy to use? <br />(n = 5328, mean: 1,8)<br />1<br />2<br />3<br />How do you rate thewholeexam? (n = 5155, mean: 3,0)<br />1<br />2<br />3<br />4<br />5<br />6<br />(1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)<br />
  31. 31. Acceptance summative e-assessment<br />26.08.2011<br />22<br />Do youtrust in theusedtechniques? <br />(n = 5268, mean: 3,0)<br />1<br />2<br />3<br />4<br />5<br />6<br />Doesitmake sense touse an electronic systemforwrittentests? (n = 5249, mean: 3,1)<br />4<br />5<br />6<br />1<br />2<br />3<br />(1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)<br />
  32. 32. Acceptance summative e-assessment<br />26.08.2011<br />23<br />Doesitmake sense touse MCQs? <br />(n = 5283, mean: 3,7)<br />1<br />2<br />3<br />4<br />5<br />6<br />Shouldtherebemorewrittenthan oral exams? (n = 5228, mean: 3,8)<br />1<br />2<br />3<br />4<br />5<br />6<br />Do youthingwrittenexamsaremor fair than oral exams? (n = 5217, mean: 3,1)<br />1<br />2<br />3<br />4<br />5<br />6<br />(1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)<br />
  33. 33. Post-review<br />Statisticsforeach item<br />26.08.2011<br />24<br />
  34. 34. Post-review<br />Statisticsforthewholeexam<br />26.08.2011<br />25<br />
  35. 35. Post-review<br />Log-files<br />26.08.2011<br />26<br />
  36. 36. Post-review<br />Item-database<br />26.08.2011<br />27<br />
  37. 37. Pyramide ofcompetencies (MILLER)<br />26.08.2011<br />28<br />Perfomance„in vivo“:<br />MiniCEX (undercov. stand. patients)<br />Does<br />Performance „in vitro“:<br />OSCE (stand. patients)<br />Shows<br />Prozedurale knowledge:<br />Key-feature, oral exams<br />Knows how<br />Deskriptive/ elaborativeKnowledge:<br />MCQ, oral exams<br />Knows<br />MILLER GE (1990): The Assessment of Clinical Skills/Competence/Performance. AcadMed 65, 63-67 <br />
  38. 38. Proceduraleknowledge<br />Useofmedia<br />26.08.2011<br />29<br />
  39. 39. Proceduraleknowledge<br />Picture analysis<br />26.08.2011<br />30<br />
  40. 40. Proceduraleknowledge/skills<br />Botany „o-test“<br />26.08.2011<br />31<br />E-exampartlywith„peer-generated“ MCQ<br />Identification-exercise in theuniversity´sbotanicalgarden<br />ABOLING S, WINDT K-H, POHL D, EHLERS JP: Lehr- und Prüfungsmethoden im Fach veterinärmedizinische Botanik mit besonderer Berücksichtigung des Konzepts o-test. ZFHE 6/1, 19-33<br />
  41. 41. Last days….<br />26.08.2011<br />32<br />Histology<br />Botany<br />
  42. 42. Key-featureassessment<br />26.08.2011<br />33<br />Answer question 1<br />Answer question 2<br />Continuationvignette<br />Continuationvignette<br />Vignette<br />Question 3<br />Question 1<br />Question 2<br />SCHAPER E, FISCHER MR, TIPOLD A, EHLERS JP: Fallbasiertes, elektronisches Lernen und Prüfen in der Tiermedizin - auf der Suche nach einer Alternative zu Multiple-Choice Prüfungen. Tierärztl. Umschau 66, 261-268<br />
  43. 43. Key-featureexample<br />26.08.2011<br />34<br />11 KFQ, 40 examinees -> Cronbach´sα = 0,68<br />
  44. 44. VP-Assessment<br />Vetsuisse-Faculty University Bern European School ofAdvancedVeterinary Studies<br />Examinationswith CASUS<br />About 15-20 cards per case<br />2-5 cases per exam<br />26.08.2011<br />35<br />
  45. 45. Questionformats<br />MCQ TypeA und PickN<br />Short Answer<br />Long Menu<br />Picture analysis<br />(fill-in-the-blank)<br />(drag ´n drop)<br />
  46. 46. Group work - creation<br />Create an own KF-item in a group!<br /><ul><li>Groups of 3-5 members
  47. 47. Time: about 15 minutes</li></ul>26.08.2011<br />37<br />
  48. 48. Whatis an ideal exam?<br />
  49. 49. Resultsofformerbrainstormings<br />Clinicians (TiHo Hannover)<br /><ul><li>Writtentheoreticalpart + oral practicalpart („blendedassessment“)
  50. 50. Definedlistoflearninggoals
  51. 51. Congruenceoflearning/teachingandassessment</li></ul>26.08.2011<br />39<br />
  52. 52. Resultsofformerbrainstormings<br />Educational mediascientists (eDay 2010)<br /><ul><li>No grades
  53. 53. Just portfolios
  54. 54. Ifexams: just unheralded</li></ul>26.08.2011<br />40<br />
  55. 55. Resultsofformerbrainstormings<br />VeterinaryEducators (DVG/GMA)<br /><ul><li>Competency- andaction-oriented (practical, writtenand oral, groupofexaminers)
  56. 56. Time neartolectures
  57. 57. Objective (w/o fear)
  58. 58. Interdisciplinary, broadknowledge
  59. 59. Feedback forteachersandlearners
  60. 60. Progress test, OSCE, kea-feature, portfolio
  61. 61. Noexams – evaluationduringcourses</li></ul>26.08.2011<br />41<br />
  62. 62. Conclusions…<br />E-assessmentispossible!<br />Regardtheir potential!(andcreateyourvision)<br />Develop e-assessmentfurther!<br />(don´tonlycopywrittentests)<br />26.08.2011<br />42<br />
  63. 63. Thankyouforyourattention!<br />26.08.2011<br />43<br />Contact<br />Dr. Jan P. Ehlers, M.A.<br />E-learningconsultant<br />University ofVeterinaryMedicine Hannover<br />Bünteweg 2 – D-30559 Hannover<br />+49-511-953-8054<br />jan.ehlers@tiho-hannover.de<br />www.tiho-hannover.de<br />
  64. 64. Going on with VPs…<br />Groups in NOVICE todiscussabout e-assessment:<br /><ul><li>ViEW (Assessment)
  65. 65. Veterinarystudents
  66. 66. ICT education</li></ul>26.08.2011<br />44<br />

×