Download as PDF, PPTX
![20Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ constructive alignment
“In constructive alignment, we start with the outcomes we intend students to learn, and align
teaching and assessment to those outcomes.
The outcome statements contain a learning activity, a verb, that students need to perform to best
achieve the outcome, such as “apply expectancy-value theory of motivation”, or “explain the
concept of … “. That verb says what the relevant learning activities are that the students need to
undertake in order to attain the intended learning outcome.
Learning is constructed by what activities the students carry out; learning is about what they do,
not about what we teachers do. Likewise, assessment is about how well they achieve the intended
outcomes, not about how well they report back to us what we have told them. […]
Constructive alignment can be used for individual courses, for degree programmes, and at the
institutional level, for aligning all teaching to graduate attributes.”
Source: http://www.johnbiggs.com.au/academic/constructive-alignment/](https://image.slidesharecdn.com/laubuschqualityassurancelargescaleassessments2-160607133843/75/Quality-Assurance-in-lagre-scale-e-Assessments-20-2048.jpg)
More Related Content
![[EADTU-ENQA PLA] Recommendations for QA of e-learning](https://cdn.slidesharecdn.com/ss_thumbnails/recommendationsforqaofe-learningestherhuertas-170925075532-thumbnail.jpg?width=640&height=640&fit=bounds)
![[EADTU-ENQA PLA] The E-xcellence QA methodology: lessons learned over ten yea...](https://cdn.slidesharecdn.com/ss_thumbnails/thee-xcellenceqamethodologylessonslearnedovertenyearsofdevelopmentandimplementationjonrosewelletal-170925075106-thumbnail.jpg?width=640&height=640&fit=bounds)
![[EADTU-ENQA PLA] Introduction PLA: Quality Assurance of Blended and Online Pr...](https://cdn.slidesharecdn.com/ss_thumbnails/eadtu-enqapeerlearningactivityintroductiongeorgeubachs-170925080051-thumbnail.jpg?width=640&height=640&fit=bounds)
![[EADTU-ENQA PLA] Anaylisis of questionnaires](https://cdn.slidesharecdn.com/ss_thumbnails/anaylisisofquestionnairesplajonrosewell-170925083917-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Quality Assurance in lagre scale e-Assessments
Quality Assurance in lagre scale e-Assessments
- 1. 1Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 QUALITY ASSURANCE IN LARGE SCALE E-ASSESSMENTS Prof. Dr. Heinz-Werner Wollersheim
- 2. 2Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 Content 1) Quality assurance: more than a technical problem 2) Aspects of quality assurance in MCA 3) Shortcut: important terms 4) Quality assurance process in MCA 5) Prospects
- 3. 3Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem
- 4. 4Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) More than a technical problem quality management quality assurance analysis current state implementing improvements documentation measuring effects 1. detecting & making aware processes 2. detecting & fixing frictions
- 5. 5Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem optimization circle of quality development quality assurance analysis current state implementing improvements documentation measuring effects
- 6. 6Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem quality management Goals to achieve: Doing the right things. (effectiveness) Doing the things in the right manner. (efficiency) Doing the things at the right time. (efficiency) „Our mission is to do the right things right at the right time.“
- 7. 7Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem Beyond technical “controlmania“ § technical control is working just in addition to social control (not as a substitute) § reaching high acceptance: university students, colleagues § needs lots of communication: The right things done right at the right time? § But: Assessing verifiable objectives is indicating a shift from holistic „education“ to specific „training“
- 8. 8Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem Doing the right things: validity kinds of validity relevance determining/ indicators content validity best way to operationalize contents ratings by experts construct validity Does the test measure the intended learning outcomes and competences? a) convergent validity (data of tests designed to measure similar competences are highly correlating) b) divergent validity (data of tests designed to measure different competences are lowly correlating) criterion validity correlation between measurement instrument and empirical criteria a) diagnostic validity a) prognostic validity
- 9. 9Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 1) Quality assurance: more than a technical problem Academic teaching: typical issues § common practice at the high schools and universities: modules and courses are designed at most § challenge: developing quality assured assessments in this context § validity reduced to content validity § Modified Constructive Alignment (MCA)
- 10. 10Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 2) Aspects of quality assurance in MCA
- 11. 11Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 2) Aspects of quality assurance in MCA on § objectivity: implementing objectivity objectivity of analysis § validity: content validity à ratings by experts à assessement plan (blueprint) § reliability: internal consistency à consistency analysis
- 12. 12Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 2) Aspects of quality assurance in MCA on analysis of items: § empirical difficulty: index of difficulty 20 ≤ Pi ≤ 80 § discrimination: coefficient of discrimination r ≥ .20
- 13. 13Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 2) Aspects of quality assurance in MCA § objectivity: objectivity of implementation objectivity of analysis § validity: content validity à ratings by experts, à assessement plan (blueprint) § reliability: internal consistency à consistency analysis § empirical difficulty: index of difficulty à 20 ≤ Pi ≤ 80 § discrimination: coefficient of discrimination à r ≥ .20
- 14. 14Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms
- 15. 15Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § selected response – selected response: multiple possible answers, choosing the right one(s) – legal term (Germany): Antwort-Wahl-Verfahren – terms that describes the way to give an answer: multiple-choice question type (MCQ), matching, sequence, hotspot – assessment question: item
- 16. 16Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § item “The smallest separately identified question or task within an assessment plus, its associated information (for example mark scheme, curriculum reference, media content, performance information etc), usually a single objective question. Distinguished from a ‘question’, which may be a longer and less- objective task but often used synonymously.” (Qualifications and Curriculum Authority 2007, 107)
- 17. 17Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § selected response “The selected-response item format is the best choice for test developers interested in efficient, effective measurement of cognitive achievement or ability.” (Downing 2006, 287)
- 18. 18Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § item structure Das Bild kann derzeit nicht angezeigt werden.high quality structure item stem vignette and question options correct answer and several incorrect answer (distractors) more detailed stem short stem (A) short options (A) long options avoid (B) (C) (D) (E) (B) (C) (D) (E)
- 19. 19Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § constructive alignment: the german view Ø „Beim 'Constructive-Alignment'-Konzept geht es im Kern darum, dass die intendierten Outcomes des Lernprozesses klar definiert und den Studierenden explizit verdeutlicht werden und die Prüfungs- und Lernaktivitäten stringent auf die Learning Outcomes abgestimmt werden.“ (Schaper 2012, 62) à In nuce C.A. means coherence between learning outcomes, assessment and learning process
- 20. 20Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § constructive alignment “In constructive alignment, we start with the outcomes we intend students to learn, and align teaching and assessment to those outcomes. The outcome statements contain a learning activity, a verb, that students need to perform to best achieve the outcome, such as “apply expectancy-value theory of motivation”, or “explain the concept of … “. That verb says what the relevant learning activities are that the students need to undertake in order to attain the intended learning outcome. Learning is constructed by what activities the students carry out; learning is about what they do, not about what we teachers do. Likewise, assessment is about how well they achieve the intended outcomes, not about how well they report back to us what we have told them. […] Constructive alignment can be used for individual courses, for degree programmes, and at the institutional level, for aligning all teaching to graduate attributes.” Source: http://www.johnbiggs.com.au/academic/constructive-alignment/
- 21. 21Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 3) Shortcut: important terms § learning outcomes Ø “Learning outcomes describe what a learner is expected to know, understand and be able to do after successful completion of a process of learning.” (ECTS Users‘ Guide, Europäische Gemeinschaft 2009, 11)
- 22. 22Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA
- 23. 23Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA § focus:
- 24. 24Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA 2 dimensions of quality assurance planningmodelfor didacticsinhigher education constructive alignment analysis evaluation process stages development implementation
- 25. 25Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA Planning stages according to constructive alignment learning outcomes § defining intended learning outcomes before starting the course § align teaching and assessment to those outcomes designing tests § developing assessment according to intended learning outcomes § adapting those outcomes if necessary designing learning process § designing learning process according to intended learning outcomes after developing assessment 1 2 3 learning outcomes learning process and learning activities assessment 1 2 3
- 26. 26Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA Workflow: Creating e-assessements according to constructive alignment Das Bild kann derzeit nicht angezeigt werden. planninglevels modul course (seperate course unit) modul workload/ ECTS contentstrucutredesign of the course topicofacourse indexing of topics and contents previous knowledge competence descriptions DQR/ HQR issues to be considered learningoutcomesthat arerelavantfor assessments (ifnecessary) formatandmethod ofassessment assessmentplan (blueprint)oftargetstate (contents,performancelevels, typesoftestitems) blueprint ofcurrentstate peer-reviewprocess creatingitems creatingtestitempools designing tests compiling tests creatingtests planningstages topicareas specificcontents learningoutcomes
- 27. 27Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA 2 dimensions of quality assurance planningmodelfor didacticsinhigher education constructive alignment analysis evaluation process stages development implementation specific implementation?
- 28. 28Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA simplified process model for e-assessment developing an e-exam implementing an e-exam adapting an e-exam analysing and evaluating an e-exam
- 29. 29Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 developing an e-exam implementing an e-exam adapting an e-exam analysing and evaluating an e-exam § constructive alignment (Biggs and Tang 2007) § taxonomy for learning, teaching, and assessing (Anderson and Krathwohl 2001) § learning outcomes § assessment plan (blueprint) § scoring model § content and form of test items § peer-review analysis of tests and items § empirical difficulty § discrimination § reliability renormalization § practice e-exam session § instructions for working with test items § risk management 2 levels: § content criteria for organizing the course and developing the tests § ideas for innovating the electronical assessments system
- 30. 30Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 § knowledge of constructive alignment (Biggs and Tang 2007) and the taxonomy for learning, teaching, and assessing (Anderson and Krathwohl 2001) § knowledge of the form and content of correct test items § different work assistance tools to organize the developing process § defining learning outcomes for courses § designing the assessment plan (blueprint): number of items, performance levels, forms of knowledge, topics; ensuring one-dimensionality § calculating guessing probability § organizing the item developing (work assistance tools) § organizing the peer review process § creating tests: selecting items according to the assessment plan (blueprint) equal opportunities in case of more than one test in course Entwicklung der E-Klausur developing an e-exam
- 31. 31Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 implementing an e-exam § training version of the e-exam session: university students get to know about test procedure (e.g. navigation) and kinds of item types and designs to become familiar with undertaking e-exams § instructions for working with test items § transparency of scoring model, e.g. getting marks according to all-or-nothing-principle or for each part of the right answer § risk management: controling examination server, back-up server, log files, bug documentation
- 32. 32Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 analysing and evaluating an e-Exam analysis of items and test: item level: § empirical difficulty: index of difficulty 20 ≤ ≤ 80 § selectivity (corrected item-total discrimination): coefficient of item discrimination r ≥ .20 test level: § reliability: internal consistency à consistency analysis .80 ≤ KR 20 ≤ .90 renormalization
- 33. 33Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 4) Quality assurance process in MCA simplified process model for e-assessment developing an e-exam implementing an e-exam adapting an e-exam analysing and evaluating an e-exam
- 34. 34Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects
- 35. 35Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects objectivity validity reliability empirical difficulty selectivity / item-total discrimination security against miscalculation scaling dealing with guessing of university students dealing with different valuate modes in item analysis quality criterion/ quality feature actually solved implemented in ILIAS 4.3.7
- 36. 36Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects § descriptive statistics + inferential statistics are necessary to verification the one-dimensionality of the items „Die einzig angemessene Umgehensweise mit dem Problem des Ratens ist der Einsatz ganz spezieller Methoden der IRT, die bei der Schätzung des jeweils gesuchten Fähigkeitsparameters einer Person aufgabenspezifisch das faktische Erfolgsausmaß beim Versuch des Lösungerratens mit ein- kalkulieren (d.s. insbesondere das 3-PL Modell und das Difficulty plus Guessing PL Modell; vgl. wieder Kubinger, 2009).“ (Kubinger 2014, 170) § Beispiel: Verrechnungsregel im Rahmen der Skalierung (z.B. Anzahl gelöster Aufgaben) à sollte mit Hilfe der Item-Response-Theorie auf empirische Angemessenheit geprüft werden
- 37. 37Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects dealing with guessing – proposal for solution I: § guessing probability included in the calculation of scoring model à issue: different behaviour of university students in relation to guessing
- 38. 38Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects dealing with guessing – proposal for solution II: § increasing the number of distracors § “1 of 6“, “1 of 7“, “1 of 8“ § “x of 5“, scoring model with all-or-nothing-principle à issues: 1. valid assessment required appropriate distracors (e.g. plausible, homogeneous) à analysing distractor quality 2. scoring model with all-or-nothing-principle (applied to “x of 5“) assumes that partly knowledge is not enough (cf. Kubinger 2014)
- 39. 39Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 5) Prospects minimum goals to reach validity creating workflow to managing the working process, observing the standards, improving the usability
- 40. 40Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 literature reference Amtsblatt der Europäische Union (AblEU) Nr. 2008/C 111/01 v. 6.5.2008. Source: http://eur-lex.europa.eu/LexUriServ/LexUri-Serv.do?uri=OJ:C:2008:111:0001:0007:DE:PDF Downing, Steven M. (2006): Selected-Response Item Formats in Test Development. In: Downing, Steven M. / Haladyna, Thomas M.: Handbook of Test Development. Mahwah, N.J., S. 287-301. Europäische Gemeinschaft (2009): ECTS-Leitfaden. Source: http://ec.europa.eu/education/tools/docs/ects-guide_de.pdf Fisseni, H.-J. (1990): Lehrbuch der psychologischen Diagnostik. Göttingen: Hogrefe. Kubinger, K. D. (2009): Psychologische Diagnostik - Theorie und Praxis psychologischen Diagnostizierens. Göttingen: Hogrefe. Kubinger, K. D. (2014): Kubinger, Klaus D. (2014): Gutachten zur Erstellung „gerichtsfester” Multiple-Choice-Prüfungsaufgaben. In: Psychologische Rundschau 65 (3), S. 169–178. Lienert, G. A. & Raatz, U. (1998): Testaufbau und Testanalyse. Weinheim: Beltz PVU. Schaper, N. (2012): Fachgutachten zur Kompetenzorientierung in Studium und Lehre. Source: http://www.hrk-nexus.de/fileadmin/redaktion/hrk-nexus/07-Downloads/07-02-Publikationen/fachgutachten_kompetenzorientierung.pdf Qualifications and Curriculum Authority (2007): e-Assessment. Guide to effective practice. Soruce: http://www.e-assessment.com/wp-content/uploads/2014/08/e-assessment_-_guide_to_effective_practice_full_version.pdf
- 41. 41Prof. Dr. H.-W.Wollersheim Quality assurance in Large Scale E-Assessments Tempus Workshop Laubusch 2016 THANK YOU FOR LISTENING. Prof. Dr. Heinz-Werner Wollersheim Leipzig University Institute of Educational Sciences Chair for General Pedagogy wollersheim@uni-leipzig.de