The document discusses quality assurance in large scale e-assessments. It outlines a quality assurance process that involves (1) planning assessments according to constructive alignment principles by defining learning outcomes and designing an assessment blueprint, (2) developing assessments by creating test items and compiling tests, and (3) analyzing and evaluating assessments by analyzing item-level metrics like difficulty and discrimination and test-level reliability. The process aims to ensure assessments are valid, objective, and reliable. Quality assurance is about more than just technical issues - it also requires communication and buy-in from students and faculty.

1. 1Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
QUALITY ASSURANCE
IN LARGE SCALE E-ASSESSMENTS
Prof. Dr. Heinz-Werner Wollersheim

2. 2Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
Content
1) Quality assurance: more than a technical problem
2) Aspects of quality assurance in MCA
3) Shortcut: important terms
4) Quality assurance process in MCA
5) Prospects

3. 3Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem

4. 4Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) More than a technical problem
quality
management
quality
assurance
analysis
current state
implementing
improvements
documentation
measuring
effects
1. detecting & making aware
processes
2. detecting & fixing frictions

5. 5Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem
optimization
circle
of
quality
development
quality
assurance
analysis
current state
implementing
improvements
documentation
measuring
effects

6. 6Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem
quality
management
Goals to achieve:
Doing the right things. (effectiveness)
Doing the things in the right manner. (efficiency)
Doing the things at the right time. (efficiency)
„Our mission is to do the right things
right at the right time.“

7. 7Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem
Beyond technical “controlmania“
§ technical control is working just in addition to social control (not as
a substitute)
§ reaching high acceptance: university students, colleagues
§ needs lots of communication: The right things done right at the right
time?
§ But: Assessing verifiable objectives is indicating a shift from holistic
„education“ to specific „training“

8. 8Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem
Doing the right things: validity
kinds of validity relevance determining/ indicators
content validity best way to operationalize contents ratings by experts
construct validity
Does the test measure the intended
learning outcomes and competences?
a) convergent validity
(data of tests designed to measure
similar competences are
highly correlating)
b) divergent validity
(data of tests designed to measure
different competences are
lowly correlating)
criterion validity
correlation between measurement
instrument and empirical criteria
a) diagnostic validity
a) prognostic validity

9. 9Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
1) Quality assurance: more than a technical problem
Academic teaching: typical issues
§ common practice at the high schools and universities:
modules and courses are designed at most
§ challenge:
developing quality assured assessments in this context
§ validity reduced to content validity
§ Modified Constructive Alignment (MCA)

10. 10Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
2) Aspects of quality assurance in MCA

11. 11Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
2) Aspects of quality assurance in MCA on
§ objectivity: implementing objectivity
objectivity of analysis
§ validity: content validity à ratings by experts
à assessement plan (blueprint)
§ reliability: internal consistency à consistency analysis

12. 12Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
2) Aspects of quality assurance in MCA on
analysis of items:
§ empirical difficulty: index of difficulty 20 ≤ Pi ≤ 80
§ discrimination: coefficient of discrimination r ≥ .20

13. 13Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
2) Aspects of quality assurance in MCA
§ objectivity: objectivity of implementation
objectivity of analysis
§ validity: content validity à ratings by experts,
à assessement plan (blueprint)
§ reliability: internal consistency à consistency analysis
§ empirical difficulty: index of difficulty à 20 ≤ Pi ≤ 80
§ discrimination: coefficient of discrimination à r ≥ .20

14. 14Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms

15. 15Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ selected response
– selected response: multiple possible answers,
choosing the right one(s)
– legal term (Germany): Antwort-Wahl-Verfahren
– terms that describes the way
to give an answer: multiple-choice question type (MCQ),
matching, sequence, hotspot
– assessment question: item

16. 16Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ item
“The smallest separately identified question or task within an assessment plus,
its associated information (for example mark scheme, curriculum reference,
media content, performance information etc), usually a single objective
question. Distinguished from a ‘question’, which may be a longer and less-
objective task but often used synonymously.”
(Qualifications and Curriculum Authority 2007, 107)

17. 17Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ selected response
“The selected-response item format is the best choice
for test developers interested in efficient, effective measurement
of cognitive achievement or ability.”
(Downing 2006, 287)

18. 18Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ item structure
Das Bild kann derzeit nicht angezeigt werden.high quality structure
item stem
vignette and question
options
correct answer
and
several incorrect answer
(distractors)
more detailed stem
short stem
(A) short options (A) long options
avoid
(B)
(C)
(D)
(E)
(B)
(C)
(D)
(E)

19. 19Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ constructive alignment: the german view
Ø „Beim 'Constructive-Alignment'-Konzept geht es im Kern darum, dass
die intendierten Outcomes des Lernprozesses klar definiert und den
Studierenden explizit verdeutlicht werden und die Prüfungs- und
Lernaktivitäten stringent auf die Learning Outcomes abgestimmt
werden.“
(Schaper 2012, 62)
à In nuce C.A. means coherence between
learning outcomes,
assessment and
learning process

20. 20Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ constructive alignment
“In constructive alignment, we start with the outcomes we intend students to learn, and align
teaching and assessment to those outcomes.
The outcome statements contain a learning activity, a verb, that students need to perform to best
achieve the outcome, such as “apply expectancy-value theory of motivation”, or “explain the
concept of … “. That verb says what the relevant learning activities are that the students need to
undertake in order to attain the intended learning outcome.
Learning is constructed by what activities the students carry out; learning is about what they do,
not about what we teachers do. Likewise, assessment is about how well they achieve the intended
outcomes, not about how well they report back to us what we have told them. […]
Constructive alignment can be used for individual courses, for degree programmes, and at the
institutional level, for aligning all teaching to graduate attributes.”
Source: http://www.johnbiggs.com.au/academic/constructive-alignment/

21. 21Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
3) Shortcut: important terms
§ learning outcomes
Ø “Learning outcomes describe what a learner is expected to know,
understand and be able to do after successful completion of a process
of learning.”
(ECTS Users‘ Guide, Europäische Gemeinschaft 2009, 11)

22. 22Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA

23. 23Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
§ focus:

24. 24Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
2 dimensions of quality assurance
planningmodelfor
didacticsinhigher
education
constructive alignment
analysis evaluation
process stages
development implementation

25. 25Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
Planning stages according to constructive alignment
learning outcomes
§ defining intended learning outcomes
before starting the course
§ align teaching and assessment to those
outcomes
designing tests
§ developing assessment according to
intended learning outcomes
§ adapting those outcomes if necessary
designing learning process
§ designing learning process according to
intended learning outcomes after
developing assessment
1
2
3
learning
outcomes
learning process
and
learning activities
assessment
1
2
3

26. 26Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
Workflow:
Creating e-assessements according to constructive alignment
Das Bild kann derzeit nicht angezeigt werden.
planninglevels
modul
course (seperate course unit)
modul
workload/
ECTS
contentstrucutredesign
of the course
topicofacourse
indexing
of topics and
contents
previous
knowledge
competence
descriptions
DQR/
HQR
issues to be considered
learningoutcomesthat
arerelavantfor
assessments
(ifnecessary)
formatandmethod
ofassessment
assessmentplan
(blueprint)oftargetstate
(contents,performancelevels,
typesoftestitems)
blueprint
ofcurrentstate
peer-reviewprocess
creatingitems
creatingtestitempools
designing tests compiling tests
creatingtests
planningstages
topicareas
specificcontents
learningoutcomes

27. 27Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
2 dimensions of quality assurance
planningmodelfor
didacticsinhigher
education
constructive alignment
analysis evaluation
process stages
development implementation
specific implementation?

28. 28Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
simplified process model for e-assessment
developing
an e-exam
implementing
an e-exam
adapting
an e-exam
analysing and
evaluating
an e-exam

29. 29Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
developing
an e-exam
implementing
an e-exam
adapting
an e-exam
analysing and
evaluating
an e-exam
§ constructive alignment (Biggs and Tang 2007)
§ taxonomy for learning, teaching, and assessing
(Anderson and Krathwohl 2001)
§ learning outcomes
§ assessment plan (blueprint)
§ scoring model
§ content and form of test items
§ peer-review
analysis of tests and items
§ empirical difficulty
§ discrimination
§ reliability
renormalization
§ practice e-exam session
§ instructions for working
with test items
§ risk management
2 levels:
§ content criteria for organizing the
course and developing the tests
§ ideas for innovating the electronical
assessments system

30. 30Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
§ knowledge of constructive alignment (Biggs and Tang 2007) and the taxonomy for
learning, teaching, and assessing (Anderson and Krathwohl 2001)
§ knowledge of the form and content of correct test items
§ different work assistance tools to organize the developing process
§ defining learning outcomes for courses
§ designing the assessment plan (blueprint):
number of items, performance levels, forms of knowledge, topics;
ensuring one-dimensionality
§ calculating guessing probability
§ organizing the item developing (work assistance tools)
§ organizing the peer review process
§ creating tests:
selecting items according to the assessment plan (blueprint)
equal opportunities in case of more than one test in course
Entwicklung
der E-Klausur
developing
an e-exam

31. 31Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
implementing
an e-exam
§ training version of the e-exam session: university students get to know about test
procedure (e.g. navigation) and kinds of item types and designs to become familiar with
undertaking e-exams
§ instructions for working with test items
§ transparency of scoring model, e.g. getting marks according to all-or-nothing-principle or
for each part of the right answer
§ risk management: controling examination server, back-up server, log files, bug documentation

32. 32Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
analysing and
evaluating
an e-Exam
analysis of items and test:
item level:
§ empirical difficulty: index of difficulty 20 ≤ ≤ 80
§ selectivity (corrected item-total discrimination):
coefficient of item discrimination r ≥ .20
test level:
§ reliability: internal consistency à consistency analysis .80 ≤ KR 20 ≤ .90
renormalization

33. 33Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
4) Quality assurance process in MCA
simplified process model for e-assessment
developing
an e-exam
implementing
an e-exam
adapting
an e-exam
analysing and
evaluating
an e-exam

34. 34Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects

35. 35Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects
objectivity
validity
reliability
empirical difficulty
selectivity / item-total discrimination
security against miscalculation
scaling
dealing with guessing of university students
dealing with different valuate modes in
item analysis
quality criterion/ quality feature
actually
solved
implemented
in
ILIAS 4.3.7

36. 36Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects
§ descriptive statistics + inferential statistics are necessary to
verification the one-dimensionality of the items
„Die einzig angemessene Umgehensweise mit dem Problem des Ratens ist
der Einsatz ganz spezieller Methoden der IRT, die bei der Schätzung
des jeweils gesuchten Fähigkeitsparameters einer Person aufgabenspezifisch
das faktische Erfolgsausmaß beim Versuch des Lösungerratens mit ein-
kalkulieren (d.s. insbesondere das 3-PL Modell und das Difficulty plus
Guessing PL Modell; vgl. wieder Kubinger, 2009).“
(Kubinger 2014, 170)
§ Beispiel: Verrechnungsregel im Rahmen der Skalierung
(z.B. Anzahl gelöster Aufgaben)
à sollte mit Hilfe der Item-Response-Theorie auf
empirische Angemessenheit geprüft werden

37. 37Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects
dealing with guessing – proposal for solution I:
§ guessing probability included in the calculation of scoring model
à issue: different behaviour of university students
in relation to guessing

38. 38Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects
dealing with guessing – proposal for solution II:
§ increasing the number of distracors
§ “1 of 6“, “1 of 7“, “1 of 8“
§ “x of 5“, scoring model with all-or-nothing-principle
à issues:
1. valid assessment required appropriate distracors
(e.g. plausible, homogeneous)
à analysing distractor quality
2. scoring model with all-or-nothing-principle (applied to “x of 5“)
assumes that partly knowledge is not enough
(cf. Kubinger 2014)

39. 39Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
5) Prospects
minimum goals to reach validity
creating workflow to
managing the working process,
observing the standards,
improving the usability

40. 40Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
literature reference
Amtsblatt der Europäische Union (AblEU) Nr. 2008/C 111/01 v. 6.5.2008.
Source: http://eur-lex.europa.eu/LexUriServ/LexUri-Serv.do?uri=OJ:C:2008:111:0001:0007:DE:PDF
Downing, Steven M. (2006): Selected-Response Item Formats in Test Development. In: Downing, Steven M. / Haladyna, Thomas
M.: Handbook of Test Development. Mahwah, N.J., S. 287-301.
Europäische Gemeinschaft (2009): ECTS-Leitfaden.
Source: http://ec.europa.eu/education/tools/docs/ects-guide_de.pdf
Fisseni, H.-J. (1990): Lehrbuch der psychologischen Diagnostik. Göttingen: Hogrefe.
Kubinger, K. D. (2009): Psychologische Diagnostik - Theorie und Praxis psychologischen Diagnostizierens. Göttingen: Hogrefe.
Kubinger, K. D. (2014): Kubinger, Klaus D. (2014): Gutachten zur Erstellung „gerichtsfester” Multiple-Choice-Prüfungsaufgaben.
In: Psychologische Rundschau 65 (3), S. 169–178.
Lienert, G. A. & Raatz, U. (1998): Testaufbau und Testanalyse. Weinheim: Beltz PVU.
Schaper, N. (2012): Fachgutachten zur Kompetenzorientierung in Studium und Lehre.
Source: http://www.hrk-nexus.de/fileadmin/redaktion/hrk-nexus/07-Downloads/07-02-Publikationen/fachgutachten_kompetenzorientierung.pdf
Qualifications and Curriculum Authority (2007): e-Assessment. Guide to effective practice.
Soruce: http://www.e-assessment.com/wp-content/uploads/2014/08/e-assessment_-_guide_to_effective_practice_full_version.pdf

41. 41Prof. Dr. H.-W. Wollersheim
Quality assurance
in Large Scale E-Assessments
Tempus
Workshop
Laubusch 2016
THANK YOU FOR LISTENING.
Prof. Dr. Heinz-Werner Wollersheim
Leipzig University
Institute of Educational Sciences
Chair for General Pedagogy
wollersheim@uni-leipzig.de