The document discusses three main challenges of modern computer-based assessment: usability, scoring, and assessing digital natives. It summarizes the COGSIM test development process which included multiple usability studies and pilot tests to improve the usability, identify scoring approaches, and ensure the assessment engaged digital native test-takers. The process led to enhancements in areas like the graphical user interface, how performance was measured both in terms of products and processes, and design elements to attract and maintain the interest of digital natives accustomed to technology.

1. Challenges of modern
Computer Based Assessment:
Usability, Scoring and “Digital Natives”
Sonnleitner, Pa., Brunner, M.a, Keller, U.a, Martin, R.a,
Latour, T.b, Hazotte, C.b, Mayer, H.b
a… University of Luxembourg
b… Centre de Recherche Henri Tudor
TAO-Days 2011
30.03.2011

2. What is meant by „modern“ Computer Based Assessment?
Kubinger(1995) differentiates between 2 types of CBA…
• Computerized administration of paper-pencil tests
• Tests originally developed for CBA like:
- Objective Personality Tests
(Ortner, Proyer & Kubinger, 2006)
- Complex Problem Solving Scenarios
(Greiff & Funke, 2009; Sonnleitner et al., 2010)
Advantages of the latter are manifold (Kyllonen, 2009;
Martin, 2008; Ridgway & McCusker, 2003):
• process- as well as product measures
• assessment of more complex cognitive abilities
• ICT-Literacy is covered
•…

3. Typical CBA-item…
“Raven Matrices”
Source: IQ-test.dk http://iqtest.dk/

4. The COGSIM project:
Assessing General Cognitive Ability by means of
Complex Problem Solving Scenarios
developed in close collaboration with…
• Centre de Recherche Public Henri Tudor
• University of Heidelberg
Aims
• Development of computer-based assessment of GCA
based on complex problem solving scenarios
• Investigation of psychometric quality and fairness
of the assessment with a large, representative sample of students
• Free distribution of the assessment (via open-source licence)

5. DEMO of the
Genetics Lab

6. Traditional Test Development Process (Shum, 2006):
Specify Construct
Check Literature for existing Test
Choose a Measurement Model
Write and Edit Items
2 possible
Administer and Analyse Responses Feedback-loops
Select „Best“ Items for Test
Check Reliability and Validity
Norm
Prepare Test Manual
Publish Test

7. COGSIM Test Development Process:
1st Usability Study, n = 8
Redesign & Programing
Specify Construct
Check Literature for existing Test
2nd Usability Study, n = 8
Choose a Measurement Model Modification & Programing
Write and Edit Items 4 development
1st Pilot Study, n = 59 cycles
Administer and Analyse Responses
Modification & Programing
Select „Best“ Items for Test
Check Reliability and Validity
2nd Pilot Study, n = 79
Norm
Prepare Test Manual
3rd Usability Study incl.
Focus group, n = 7
Publish Test
Modification & Programing

8. COGSIM Test Development Process:
1st Usability Study, n = 8 qualitative analysis
Redesign & Programing
2nd Usability Study, n = 8
qualitative analysis
Modification & Programing 3 main challenges
were identified:
1st Pilot Study, n = 59
quantitative analysis - Usability
Modification & Programing
- Scoring
- Digital natives
2nd Pilot Study, n = 79
3rd Usability Study incl.
Focus group, n = 7 qualitative analysis
Modification & Programing

9. These elements are interconnected…
Assessment Instrument
(Scoring of Performance)
Target Population Usability
(Digital Natives) (Instructions + GUI)

10. Challenge 1:
Usability

11. The role of Usability…
Assessment Instrument
Design,
Scoring/ Validity
Semantics, etc.
Usability
Target Population Instructions
Aim of good interface design is to reduce construct-
irrelevant variance that could be attributed to test method
(Fulcher, 2003; Messick, 1989)

12. Identifiying Usability-Problems:
Qualitative data:
1st Usability Study, n = 8 • Think-aloud protocols
• Observation protocols
• Interviews
2nd Usability Study, n = 8 + Focus group in SSUS 3
1st Pilot Study, n = 59
Quantitative data:
2nd Pilot Study, n = 79 • Usability Questionnaire incl.
- Functionality of each element
- Comprehensibility
3rd Usability Study incl. - Subjective Difficulty
Focus group, n = 7 - Attractiveness

13. Classifying Usability-Problems:
Identified Construct related:
Problem
• due to difficulty of tasks
possible change of construction rationale
Usability related – 3 categories:
• basic level
(e.g. size of letters, use of colors, …)
• medium level
(e.g. navigation within instruction/ between items,
guidance of attention, …)
• high level
(e.g. working on task, using the concept map, …)

14. „Evolution“ of some elements of the GUI:
Basic level problem – position of variable-value:
SSUS 1
Pilot study 1
Main study –
final version

15. „Evolution“ of some elements of the GUI:
Medium level problem – navigate within a task:
SSUS 1
Pilot study 1
Main study –
final version

16. „Evolution“ of some elements of the GUI:
High level problem – using the concept map:
SSUS 1
Pilot study 1
Main study –
final version

17. Correlations
Wie oft
PC-Spiele/ sc.sysex.r sc.gdk.gl sc.stars2.c
Woche? el.tot obal.tot trl.raw.tot
Wie oft Pearson Correlation 1 .076 .281 .327*
Indicators for improved usability?
PC-Spiele/Woche? Sig. (2-tailed) . .661 .083 .042
N 39 36 39 39
sc.sysex.rel.tot Pearson Correlation .076 1 .539** .347*
Pilot study 1 Sig. (2-tailed)
N Correlations
.661 . .001 .038
36 36 36 36
sc.gdk.global.tot Pearson Correlation Correlations
.281 .539** 1 .399*
Wie oft
Sig. (2-tailed) Tage proPC-Spiele/ pro .001
.083
Stunden sc.sysex.r sc.gdk.gl. sc.stars2.c .012
N Woche mit Woche? Wochemit el.tot36
39 39
obal.tot trl.raw.tot 39
Wie oft
sc.stars2.ctrl.raw.tot Pearson Correlation Computer Computerspi sc.sysex.r sc.gdk.gl sc.stars2.c
1
.327* .076
.347* .281
.399* .327*1
PC-Spiele/Woche? Sig. (2-tailed) spielen .042elen
. el.tot
.661
.038 obal.tot
.083
.012 trl.raw.tot . Summe ICT
.042
Tage pro Woche mit Pearson Correlation
N 1 .801** .138 -.017 .174 .228
39 36 39 39
Computerspielen Sig. (2-tailed) Correlation . .076 .000 .296 .897 .187 .086
sc.sysex.rel.tot is significant at the 0.05 level (2-tailed).
*. Correlation Pearson 1 .539** .347*
N 59 59 59 59 59 58
**. Correlation Sig. (2-tailed) .661 . .001 .038
Stunden pro Wochemitis significant at the 0.01 level .801**
Pearson Correlation (2-tailed). 1 .094 .048 .087 .276*
N 36 36 36 36
Computerspielen Sig. (2-tailed)
sc.gdk.global.tot Pearson Correlation .000 .281
.
.539**
.473 .715
1
.508
.399*
.035
N 59 60 60 60 60 59
Sig. (2-tailed) .083 .001 . .012
sc.sysex.rel.tot
N
Sig. (2-tailed)
Enhancement of Usability
Pearson Correlation .138
39
.094
36
1
39
.401** .307*
39
.203
sc.stars2.ctrl.raw.tot Pearson Correlation .296 .327*
.473
.347*
. .001
.399*
.016
1
.121
N 59 60 61 61 61 60
Sig. (2-tailed) .042 .038 .012 .
sc.gdk.global.tot Pearson Correlation -.017 .048 .401** 1 .544** .295*
Pilot study 2 N
Sig. (2-tailed) .897
Correlations
39 36
Correlations.001
.715
39
.
39
.000 .022
*. Correlation is significant at the 0.05 level (2-tailed).
N 59 60 61 61 61 60
**. Correlation Tage pro Tage pro
Stunden pro Stunden pro
sc.stars2.ctrl.raw.tot is significant at the 0.01 level .174
Pearson Correlation (2-tailed). .087 .307* .544** 1 .299*
Woche mit Wochemitmit
Woche Wochemit
Sig. (2-tailed)
Computer Computerspi .508
.187
Computer sc.sysex.r .016
Computerspi .000 sc.stars2.c.
sc.sysex.r
sc.gdk.gl sc.gdk.gl .020
sc.stars2.c
N spielen 59 spielen
elen 60 elen 61 obal.tot 61 trl.raw.tot
el.tot el.tot obal.tot Summe60
61 trl.raw.tot
ICT Sum
Summe ICTWoche mit Pearson Correlation 1
Tage pro
Tage pro Woche mit Pearson Correlation Pearson Correlation .228 .801** .276* .138 .801** -.017 .138
1 .203 .295* .299*
.174 -.017 1 .174
.228
Computerspielen Computerspielen
Sig. (2-tailed) Sig. (2-tailed)
Sig. (2-tailed) . .086 .000 . .035 .296 .000 .121 .022
.897 .296 .020
.187 .897 ..187
.086
N N N 59 58 5959 59 59 60 59 59 6059 60
59 59 60 59
58
StundenPearson Correlation
**. pro Wochemit Pearson Correlation
Stunden pro Wochemit Correlation is significant at the 0.01 level (2-tailed).
.801** .801**
1 .094 1 .048 .094 .087 .048 .087
.276*
Computerspielen Computerspielen
Sig. (2-tailed) Sig. (2-tailed) .000 . .473 .715 .508
*. Correlation is significant at the 0.05 level .000
(2-tailed). . .473 .715 .508 .035
N N 59 6059 60 60 60 60 60 60 59 60
sc.sysex.rel.tot sc.sysex.rel.tot Correlation
Pearson Pearson Correlation
.138 .138
.094 1 .094 .401** 1 .307* .401** .203.307*
Sig. (2-tailed) Sig. (2-tailed) .296 .296
.473 . .473 .001 . .016 .001 .121.016

18. But loosing weight is definitely positive…
Close interaction between Usability and Validity!

19. Challenge 2:
Scoring

20. Challenge 2: Scoring or what happens if we consider the process?
Traditional MC-item:
• Correct or False (Product), Process is unknown
Modern CBA-item:
• Product and Process itself gets measured
• Nearly endless possibilities to score (time, …)

21. Example Control Phase: Product or Process?
The task: to achieve certain target values within 3 steps

22. Possible Cognitions during Control Phase

23. Comparison of Product (target achievement) and
Process (way to target) Score:
Item 4:
+
Dx= -2
A X
Xt+1= Xt + 1*At + (-1)*T (with T = 1)
B - Y Dy= 0
T
Walk 1: 0,1 / 0,1/ 1,1 Product Score: 5/5
Process Score: 3/3
Overestimation of
Walk 2: 1,0 / 0,1/ 0,1 Product Score: 5/5
performance
Process Score: 2/3
Walk 3: 0,1 / 0,1/ 1,0 Product Score: 5/5
Process Score: 2/3
Walk 4: 0,1 / 0,1/ 0,0 Product Score: 3/5
Process Score: 1/3

24. Challenge 3:
Digital Natives

25. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006)
Who are they?
• generation born since 1990
• grown up in a world in which ICT is permanently available
What makes them special?
• used to process huge amounts of information
• permanent information overload – filtering strategies
• strongly rely on images and symbols
• deal with new technology in a non-linear way
(start to play before reading instructions)
• technology is there to solve them
• if problems occur, technology is blamed
• used to video games
• used to learn by discovery and by experimenting
• they posess iconic skills
(use symbols, icons and color-code to navigate)

26. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006)
Why is this important for test developers?
• want to be active from the first minute on (like in video games)
• they expect perfect functioning technology
• they expect an appealing GUI
• they are used to actively explore and learn, not being told
• most likely motivated when feel attracted by the design/
if task seems to be interesting and challenging
 e.g. static instructions are not a good idea

27. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006)
How to react?
• extensive usability studies with digital natives as experts
• ensure perfect functioning
• ensure appealing design
• keep them active and in exercises from the beginning on
• keep text to an absolute minimum
• include game-like characteristics
• explain using images/ animations rather than text
• use symbols, icons and color-code in an expected way

28. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006)
Evidence from our studies:
• arising questions during instruction phase
• ignored written information
• exercises including feedback improved understanding
• game-like characteristics were appreciated
•…
 it is wise to consider characteristics
of target population

29. These elements are interconnected…
Assessment Instrument
(Scoring of Performance)
Target Population Usability
(Digital Natives) (Instructions + GUI)

30. Modified Test Development Process for CBA:
Specify Construct Specify Target Population
Check Literature for existing Test
Choose a Measurement Model
Write and Edit Items Design of User Interface
Administer and Analyse Responses
regarding Construct regarding Usability
Select „Best“ Items for Test
Integration of 2 new feedback loops
Check Reliability and Validity
Norm
Publish Test

31. FIRE-Simulation (Brehmer, 1987)

32. Multiflux (Kröner, 2000, 2001)

33. MicroDYN (Greiff & Funke, 2009)

34. Genetics Lab (Sonnleitner et al., 2010)

35. „Take home“-messages:
When dealing with modern CBA:
- pay attention to usability and consider it
during development process
- think about more complex ways to score performance
- think about the special needs of your target population

36. Thank you!
Questions???