Your SlideShare is downloading. ×
0
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
TAO DAYS - Challenges of Modern Computer Based Assessment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

TAO DAYS - Challenges of Modern Computer Based Assessment

476

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
476
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Challenges of modern Computer Based Assessment:Usability, Scoring and “Digital Natives” Sonnleitner, Pa., Brunner, M.a, Keller, U.a, Martin, R.a, Latour, T.b, Hazotte, C.b, Mayer, H.b a… University of Luxembourg b… Centre de Recherche Henri Tudor TAO-Days 2011 30.03.2011
  • 2. What is meant by „modern“ Computer Based Assessment? Kubinger(1995) differentiates between 2 types of CBA… • Computerized administration of paper-pencil tests • Tests originally developed for CBA like: - Objective Personality Tests (Ortner, Proyer & Kubinger, 2006) - Complex Problem Solving Scenarios (Greiff & Funke, 2009; Sonnleitner et al., 2010) Advantages of the latter are manifold (Kyllonen, 2009; Martin, 2008; Ridgway & McCusker, 2003): • process- as well as product measures • assessment of more complex cognitive abilities • ICT-Literacy is covered •…
  • 3. Typical CBA-item… “Raven Matrices” Source: IQ-test.dk http://iqtest.dk/
  • 4. The COGSIM project:Assessing General Cognitive Ability by means ofComplex Problem Solving Scenariosdeveloped in close collaboration with…• Centre de Recherche Public Henri Tudor• University of HeidelbergAims • Development of computer-based assessment of GCA based on complex problem solving scenarios • Investigation of psychometric quality and fairness of the assessment with a large, representative sample of students • Free distribution of the assessment (via open-source licence)
  • 5. DEMO of theGenetics Lab
  • 6. Traditional Test Development Process (Shum, 2006): Specify Construct Check Literature for existing Test Choose a Measurement Model Write and Edit Items 2 possible Administer and Analyse Responses Feedback-loops Select „Best“ Items for Test Check Reliability and Validity Norm Prepare Test Manual Publish Test
  • 7. COGSIM Test Development Process: 1st Usability Study, n = 8 Redesign & Programing Specify Construct Check Literature for existing Test 2nd Usability Study, n = 8 Choose a Measurement Model Modification & Programing Write and Edit Items 4 development 1st Pilot Study, n = 59 cycles Administer and Analyse Responses Modification & Programing Select „Best“ Items for Test Check Reliability and Validity 2nd Pilot Study, n = 79 Norm Prepare Test Manual 3rd Usability Study incl. Focus group, n = 7 Publish Test Modification & Programing
  • 8. COGSIM Test Development Process:1st Usability Study, n = 8 qualitative analysis Redesign & Programing2nd Usability Study, n = 8 qualitative analysisModification & Programing 3 main challenges were identified: 1st Pilot Study, n = 59 quantitative analysis - UsabilityModification & Programing - Scoring - Digital natives 2nd Pilot Study, n = 793rd Usability Study incl. Focus group, n = 7 qualitative analysisModification & Programing
  • 9. These elements are interconnected… Assessment Instrument (Scoring of Performance) Target Population Usability (Digital Natives) (Instructions + GUI)
  • 10. Challenge 1: Usability
  • 11. The role of Usability… Assessment Instrument Design, Scoring/ Validity Semantics, etc. Usability Target Population Instructions Aim of good interface design is to reduce construct- irrelevant variance that could be attributed to test method (Fulcher, 2003; Messick, 1989)
  • 12. Identifiying Usability-Problems: Qualitative data: 1st Usability Study, n = 8 • Think-aloud protocols • Observation protocols • Interviews 2nd Usability Study, n = 8 + Focus group in SSUS 3 1st Pilot Study, n = 59 Quantitative data: 2nd Pilot Study, n = 79 • Usability Questionnaire incl. - Functionality of each element - Comprehensibility 3rd Usability Study incl. - Subjective Difficulty Focus group, n = 7 - Attractiveness
  • 13. Classifying Usability-Problems: Identified Construct related: Problem • due to difficulty of tasks possible change of construction rationale Usability related – 3 categories: • basic level (e.g. size of letters, use of colors, …) • medium level (e.g. navigation within instruction/ between items, guidance of attention, …) • high level (e.g. working on task, using the concept map, …)
  • 14. „Evolution“ of some elements of the GUI: Basic level problem – position of variable-value: SSUS 1 Pilot study 1 Main study – final version
  • 15. „Evolution“ of some elements of the GUI: Medium level problem – navigate within a task: SSUS 1 Pilot study 1 Main study – final version
  • 16. „Evolution“ of some elements of the GUI: High level problem – using the concept map: SSUS 1 Pilot study 1 Main study – final version
  • 17. Correlations Wie oft PC-Spiele/ sc.sysex.r sc.gdk.gl sc.stars2.c Woche? el.tot obal.tot trl.raw.tot Wie oft Pearson Correlation 1 .076 .281 .327*Indicators for improved usability? PC-Spiele/Woche? Sig. (2-tailed) . .661 .083 .042 N 39 36 39 39 sc.sysex.rel.tot Pearson Correlation .076 1 .539** .347* Pilot study 1 Sig. (2-tailed) N Correlations .661 . .001 .038 36 36 36 36 sc.gdk.global.tot Pearson Correlation Correlations .281 .539** 1 .399* Wie oft Sig. (2-tailed) Tage proPC-Spiele/ pro .001 .083 Stunden sc.sysex.r sc.gdk.gl. sc.stars2.c .012 N Woche mit Woche? Wochemit el.tot36 39 39 obal.tot trl.raw.tot 39 Wie oft sc.stars2.ctrl.raw.tot Pearson Correlation Computer Computerspi sc.sysex.r sc.gdk.gl sc.stars2.c 1 .327* .076 .347* .281 .399* .327*1 PC-Spiele/Woche? Sig. (2-tailed) spielen .042elen . el.tot .661 .038 obal.tot .083 .012 trl.raw.tot . Summe ICT .042 Tage pro Woche mit Pearson Correlation N 1 .801** .138 -.017 .174 .228 39 36 39 39 Computerspielen Sig. (2-tailed) Correlation . .076 .000 .296 .897 .187 .086 sc.sysex.rel.tot is significant at the 0.05 level (2-tailed). *. Correlation Pearson 1 .539** .347* N 59 59 59 59 59 58 **. Correlation Sig. (2-tailed) .661 . .001 .038 Stunden pro Wochemitis significant at the 0.01 level .801** Pearson Correlation (2-tailed). 1 .094 .048 .087 .276* N 36 36 36 36 Computerspielen Sig. (2-tailed) sc.gdk.global.tot Pearson Correlation .000 .281 . .539** .473 .715 1 .508 .399* .035 N 59 60 60 60 60 59 Sig. (2-tailed) .083 .001 . .012 sc.sysex.rel.tot N Sig. (2-tailed) Enhancement of Usability Pearson Correlation .138 39 .094 36 1 39 .401** .307* 39 .203 sc.stars2.ctrl.raw.tot Pearson Correlation .296 .327* .473 .347* . .001 .399* .016 1 .121 N 59 60 61 61 61 60 Sig. (2-tailed) .042 .038 .012 . sc.gdk.global.tot Pearson Correlation -.017 .048 .401** 1 .544** .295* Pilot study 2 N Sig. (2-tailed) .897 Correlations 39 36 Correlations.001 .715 39 . 39 .000 .022 *. Correlation is significant at the 0.05 level (2-tailed). N 59 60 61 61 61 60 **. Correlation Tage pro Tage pro Stunden pro Stunden pro sc.stars2.ctrl.raw.tot is significant at the 0.01 level .174 Pearson Correlation (2-tailed). .087 .307* .544** 1 .299* Woche mit Wochemitmit Woche Wochemit Sig. (2-tailed) Computer Computerspi .508 .187 Computer sc.sysex.r .016 Computerspi .000 sc.stars2.c. sc.sysex.r sc.gdk.gl sc.gdk.gl .020 sc.stars2.c N spielen 59 spielen elen 60 elen 61 obal.tot 61 trl.raw.tot el.tot el.tot obal.tot Summe60 61 trl.raw.tot ICT Sum Summe ICTWoche mit Pearson Correlation 1 Tage pro Tage pro Woche mit Pearson Correlation Pearson Correlation .228 .801** .276* .138 .801** -.017 .138 1 .203 .295* .299* .174 -.017 1 .174 .228 Computerspielen Computerspielen Sig. (2-tailed) Sig. (2-tailed) Sig. (2-tailed) . .086 .000 . .035 .296 .000 .121 .022 .897 .296 .020 .187 .897 ..187 .086 N N N 59 58 5959 59 59 60 59 59 6059 60 59 59 60 59 58 StundenPearson Correlation **. pro Wochemit Pearson Correlation Stunden pro Wochemit Correlation is significant at the 0.01 level (2-tailed). .801** .801** 1 .094 1 .048 .094 .087 .048 .087 .276* Computerspielen Computerspielen Sig. (2-tailed) Sig. (2-tailed) .000 . .473 .715 .508 *. Correlation is significant at the 0.05 level .000 (2-tailed). . .473 .715 .508 .035 N N 59 6059 60 60 60 60 60 60 59 60 sc.sysex.rel.tot sc.sysex.rel.tot Correlation Pearson Pearson Correlation .138 .138 .094 1 .094 .401** 1 .307* .401** .203.307* Sig. (2-tailed) Sig. (2-tailed) .296 .296 .473 . .473 .001 . .016 .001 .121.016
  • 18. But loosing weight is definitely positive… Close interaction between Usability and Validity!
  • 19. Challenge 2: Scoring
  • 20. Challenge 2: Scoring or what happens if we consider the process? Traditional MC-item: • Correct or False (Product), Process is unknown Modern CBA-item: • Product and Process itself gets measured • Nearly endless possibilities to score (time, …)
  • 21. Example Control Phase: Product or Process? The task: to achieve certain target values within 3 steps
  • 22. Possible Cognitions during Control Phase
  • 23. Comparison of Product (target achievement) andProcess (way to target) Score: Item 4: + Dx= -2 A X Xt+1= Xt + 1*At + (-1)*T (with T = 1) B - Y Dy= 0 T Walk 1: 0,1 / 0,1/ 1,1 Product Score: 5/5 Process Score: 3/3 Overestimation of Walk 2: 1,0 / 0,1/ 0,1 Product Score: 5/5 performance Process Score: 2/3 Walk 3: 0,1 / 0,1/ 1,0 Product Score: 5/5 Process Score: 2/3 Walk 4: 0,1 / 0,1/ 0,0 Product Score: 3/5 Process Score: 1/3
  • 24. Challenge 3:Digital Natives
  • 25. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006) Who are they? • generation born since 1990 • grown up in a world in which ICT is permanently available What makes them special? • used to process huge amounts of information • permanent information overload – filtering strategies • strongly rely on images and symbols • deal with new technology in a non-linear way (start to play before reading instructions) • technology is there to solve them • if problems occur, technology is blamed • used to video games • used to learn by discovery and by experimenting • they posess iconic skills (use symbols, icons and color-code to navigate)
  • 26. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006) Why is this important for test developers? • want to be active from the first minute on (like in video games) • they expect perfect functioning technology • they expect an appealing GUI • they are used to actively explore and learn, not being told • most likely motivated when feel attracted by the design/ if task seems to be interesting and challenging  e.g. static instructions are not a good idea
  • 27. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006) How to react? • extensive usability studies with digital natives as experts • ensure perfect functioning • ensure appealing design • keep them active and in exercises from the beginning on • keep text to an absolute minimum • include game-like characteristics • explain using images/ animations rather than text • use symbols, icons and color-code in an expected way
  • 28. Challenge 3: „Digital Natives“ (Prensky, 2001, Veen & Vracking, 2006) Evidence from our studies: • arising questions during instruction phase • ignored written information • exercises including feedback improved understanding • game-like characteristics were appreciated •…  it is wise to consider characteristics of target population
  • 29. These elements are interconnected… Assessment Instrument (Scoring of Performance) Target Population Usability (Digital Natives) (Instructions + GUI)
  • 30. Modified Test Development Process for CBA: Specify Construct Specify Target Population Check Literature for existing Test Choose a Measurement Model Write and Edit Items Design of User Interface Administer and Analyse Responses regarding Construct regarding Usability Select „Best“ Items for Test Integration of 2 new feedback loops Check Reliability and Validity Norm Publish Test
  • 31. FIRE-Simulation (Brehmer, 1987)
  • 32. Multiflux (Kröner, 2000, 2001)
  • 33. MicroDYN (Greiff & Funke, 2009)
  • 34. Genetics Lab (Sonnleitner et al., 2010)
  • 35. „Take home“-messages: When dealing with modern CBA: - pay attention to usability and consider it during development process - think about more complex ways to score performance - think about the special needs of your target population
  • 36. Thank you!Questions???

×