Perceived usability evaluation of LearningManagement Systems: a first step towardsstandardization of the System Usability ...
Perceived usability evaluation       of educational software   Perceived usability is an important requirement of    educ...
Perceived usability evaluation       of software: questionnairesQUIS Questionnaire for User Interface Satisfaction Chin et...
System Usability Scale (SUS)   The most widely adopted for    perceived usability evaluation    of software/products (Tul...
Research Goals   Examine the applicability of the SUS method in the context    of the usability evaluation of Learning Ma...
Methodology   3 studies       280 university students       LMS: Moodle       Courses on ICT & Education       Both b...
Studies OverviewStudy     Time                     N     Gender         Mean    Delivery                                  ...
SUS English and Greek        VersionItem                   Greek Version of SUS                                  English V...
Moodle-based course on ICT& Education (screenshot)                             9
Results (1/4) – Overview                                                                               Greek SUS     Engli...
Results (2/4) – Validity Analysis   Is the Greek SUS valid for LMS perceived usability evaluation?         Validity = th...
Results (3/4) – Reliability Analysis   Does the Greek SUS measure LMS perceived usability reliably?       Reliability = ...
Results (4/4) – Factor Analysis       Does the Greek SUS measure a single construct?             English SUS was found t...
Summary & Questions   Three studies involving 280 students assessing perceived    usability of an LMS were conducted in v...
Future Work & Preliminary        results   Effect of student’s characteristics (e.g. age, internet competence) on    SUS ...
Extra Slides               16
References (1/2)   A. Bangor, P. T. Kortum, and J. T. Miller, “Determining what individual SUS scores mean: adding an    ...
References (1/2)   N. Tselios, N. Avouris, and V. Komis, “The effective combination of hybrid usability methods in evalua...
Upcoming SlideShare
Loading in …5
×

Perceived Usability Evaluation of Learning Management Systems: A First Step towards Standardization of the System Usability Scale in Greek

1,649 views

Published on

Perceived usability is an important requirement of educational software, affecting greatly student’s learning effectiveness and overall learning experience. Although the System Usability Scale (SUS) questionnaire is considered as a de facto standard for perceived usability evaluation, surprisingly, few studies are reporting results from evaluations related to Learning Management Systems. In addition, given that the questionnaire comprises 10 questions presented in English, its applicability in studies involving speakers of other languages is questionable. In this paper, a first step towards standardization of a Greek version of SUS in the context of Learning Management Systems perceived usability evaluation is reported. To this end, three studies involving 280 university students in both blended and distance learning education were conducted. Analysis of the results demonstrated the validity and reliability of the Greek version of SUS.

Published in: Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,649
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • 15 λεπτά διαθέσιμος χρόνος => 12 λεπτά MAXπαρουσίαση
  • Perceived Usability Evaluation of Learning Management Systems: A First Step towards Standardization of the System Usability Scale in Greek

    1. 1. Perceived usability evaluation of LearningManagement Systems: a first step towardsstandardization of the System Usability Scale in Greek Christos Katsanos* | ckatsanos@eap.gr Nikolaos Tselios# | nitse@ece.upatras.gr Michalis Xenos* | xenos@eap.gr *Hellenic Open University # Educational Science and Early Childhood Education, University of Patras PCI 2012 | Piraeus, Greece | 5-7 Oct 2012
    2. 2. Perceived usability evaluation of educational software Perceived usability is an important requirement of educational software, affecting greatly student’s learning effectiveness and overall learning experience (Kostaras & Xenos 2011; Tselios et al. 2008) Numerous usability evaluation questionnaires designed to assess the perceived usability of a product and/or software are available at the moment (Brooke 1996; Lewis 1991, 1992, 1995; Tullis & Albert, 2008; Tullis & Stenson 2004)Surprisingly, rather few studies report usage of questionnairesto assess perceived usability of educational software 2
    3. 3. Perceived usability evaluation of software: questionnairesQUIS Questionnaire for User Interface Satisfaction Chin et al, 27 questions 1988PUEU Perceived Usefulness and Ease of Use Davis, 1989 12 questionsNAU Nielsens Attributes of Usability Nielsen, 1993 5 attributesNHE Nielsens Heuristic Evaluation Nielsen, 1993 10 heuristicsCSUQ Computer System Usability Questionnaire Lewis, 1995 19 questionsASQ After Scenario Questionnaire Lewis, 1995 3 questionsPHUE Practical Heuristics for Usability Evaluation Perlman, 13 heuristics 1997PUTQ Purdue Usability Testing Questionnaire Lin et al, 1997 100 questionsUSE USE Questionnaire Lund, 2001 30 questionsSUS System Usability Scale Brooke, 10 questions 1996). 3
    4. 4. System Usability Scale (SUS) The most widely adopted for perceived usability evaluation of software/products (Tullis & Albert, 2008; Sauro 2012) Why?  Quick and dirty (10 questions)  Standardized (valid, reliable)  Summarized score (0 to 100)  Reliable even with small sample (10~12 users)  SUS norms available 4 (Brooke, 1996)
    5. 5. Research Goals Examine the applicability of the SUS method in the context of the usability evaluation of Learning Management Systems (LMSs) Assess the validity and reliability of a Greek version of the SUS questionnaire  Non-native English speakers can confront problems in understanding SUS compared to native speakers (Finstad 2006) 5
    6. 6. Methodology 3 studies  280 university students  LMS: Moodle  Courses on ICT & Education  Both blended and distance education  Various study times (sem. start, mid, end) SUS  Between-subjects: Greek (N=221) vs English (N=59) versions  Based on self-rated knowledge of English  Greek version by 2 usability experts, native speakers Perceived usability 7-point rating [worst-imaginable, best-imaginable] Demographic questions (e.g. age, ICT skills, Moodle usage) 6
    7. 7. Studies OverviewStudy Time N Gender Mean Delivery Age method1 Sem. End 45 45 Females 21.04 Blended 187 Females,2 Sem. Start 191 20.45 Blended 4 Males 15 Females,3 Sem. Mid 44 37.20 Elearning 29 Males 247 Females, Blended (2)ALL {Sem. Start, Mid, End} 280 23.18 33 Males Elearning (1) 7
    8. 8. SUS English and Greek VersionItem Greek Version of SUS English Version of SUS Ννκίδω όηη ζα ήζεια λα ρξεζηκνπνηώ απηό ην I think that I would like to use this lessons MoodleQ1 ζύζηεκα Moodle ηνπ καζήκαηνο ζπρλά. system frequently. Βξήθα απηό ην ζύζηεκα Moodle ηνπ καζήκαηνο I found the lessons Moodle system unnecessarilyQ2 αδηθαηνιόγεηα πεξίπινθν. complex. Σθέθηεθα όηη απηό ην ζύζηεκα Moodle ηνπQ3 I thought the lessons Moodle system was easy to use. καζήκαηνο ήηαλ εύθνιν ζηε ρξήζε. Ννκίδω όηη ζα ρξεηαζηώ βνήζεηα από θάπνηνλ ηερληθό I think that I would need the support of a technicalQ4 γηα λα είκαη ζε ζέζε λα ρξεζηκνπνηήζω απηό ην person to be able to use this lessons Moodle system. ζύζηεκα Moodle ηνπ καζήκαηνο. Βξήθα ηηο δηάθνξεο ιεηηνπξγίεο ζε απηό ην ζύζηεκα I found the various functions in this lessons MoodleQ5 Moodle ηνπ καζήκαηνο θαιά νινθιεξωκέλεο. system were well integrated. Σθέθηεθα όηη ππήξρε κεγάιε αζπλέπεηα ζε απηό ην I thought there was too much inconsistency in thisQ6 ζύζηεκα Moodle ηνπ καζήκαηνο. lessons Moodle system. Φαληάδνκαη όηη νη πεξηζζόηεξνη άλζξωπνη ζα κάζνπλ I would imagine that most people would learn to useQ7 λα ρξεζηκνπνηνύλ απηό ην ζύζηεκα Moodle ηνπ this lessons Moodle system very quickly. καζήκαηνο πνιύ γξήγνξα. Βξήθα απηό ην ζύζηεκα Moodle ηνπ καζήκαηνο πνιύ I found the lessons Moodle system very cumbersomeQ8 πεξίπινθν/δύζθνιν ζηε ρξήζε. to use. Έληωζα πνιύ ζίγνπξνο/ε ρξεζηκνπνηώληαο απηό ηνQ9 I felt very confident using the lessons Moodle system. ζύζηεκα Moodle ηνπ καζήκαηνο. Χξεηάζηεθε λα κάζω πνιιά πξάγκαηα πξηλ λα I needed to learn a lot of things before I could getQ10 κπνξέζω λα μεθηλήζω κε απηό ην ζύζηεκα Moodle going with this lessons Moodle system. ηνπ καζήκαηνο.
    9. 9. Moodle-based course on ICT& Education (screenshot) 9
    10. 10. Results (1/4) – Overview Greek SUS English SUS Item Mean SD Mean SD Q1 3.34 0.95 3.59 0.79 Mean SUS Mean Perceived Students Q2 1.87 0.86 2.00 0.97 Study Score Usability Rating (N) (0-100) (1-7) Q3 3.75 0.89 3.74 0.911 45 74.50 ± 3.63 5.02 ± 0.20 Q4 1.47 0.77 1.66 0.902 191 70.48 ± 1.83 5.16 ± 0.11 3.69 0.83 3.47 0.80 Q53 44 71.48 ± 4.15 5.11 ± 0.27 Q6 1.89 1.04 2.14 0.90All 280 71.29 ± 1.51 5.13 ± 0.09 Q7 3.74 0.88 3.80 1.03Note: Numbers after the ± symbol are 95% confidence intervals for themean Q8 1.83 0.97 2.47 1.07 Q9 3.39 0.87 3.78 0.97 Q10 2.10 0.90 2.24 1.22 10
    11. 11. Results (2/4) – Validity Analysis Is the Greek SUS valid for LMS perceived usability evaluation?  Validity = the extent to which an instrument, such as a questionnaire, measures what it is intended to measure (Nunnally, 1994) Concurrent Measure 1: Concurrent Measure 2: Adjective Rating SUS English rs=0.474, p<0.01 t(278)=1.149, p=0.252 11*0.30 ~ 0.40 justify validity of questionnaires (Nunnally, 1994)
    12. 12. Results (3/4) – Reliability Analysis Does the Greek SUS measure LMS perceived usability reliably?  Reliability = the extent to which an instrument, such as a questionnaire, yields the same results under consistent conditions (Nunnally, 1994) Σθέθηεθα όηη ππήξρε κεγάιε I thought there was too much Q6 αζπλέπεηα ζε απηό ην ζύζηεκα inconsistency in this lessons 12 Moodle ηνπ καζήκαηνο. Moodle system.
    13. 13. Results (4/4) – Factor Analysis Does the Greek SUS measure a single construct?  English SUS was found to have two subscales: Learnability (Q4, Q10) and Usability (rest) (Lewis & Sauro, 2009) Factor analyses using common verimax rotation => No reliable subscales Component Component Component Item Item Item 1 2 1 2 3 1 2 3 4 Q1 0.781 -0.096 Q1 0.782 -0.099 0.032 Q1 0.668 -0.167 0.393 -0.136 Q2 0.485 0.356 Q2 0.462 0.184 0.388 Q2 0.244 0.139 0.740 -0.004 Q3 0.652 0.339 Q3 0.645 0.431 0.033 Q3 0.668 0.381 0.160 0.005 Q4 0.124 0.648 Q4 0.108 0.736 0.099 Q4 0.143 0.726 0.115 0.036 Q5 0.642 0.293 Q5 0.626 0.208 0.259 Q5 0.670 0.173 0.136 0.304 Q6 0.032 0.537 Q6 -0.015 0.057 0.836 Q6 0.060 0.098 0.101 0.953 Q7 0.648 0.107 Q7 0.654 0.291 -0.168 Q7 0.752 0.241 -0.100 -0.046 Q8 0.280 0.598 Q8 0.244 0.341 0.567 Q8 0.056 0.321 0.719 0.197 Q9 0.812 0.078 Q9 0.799 -0.053 0.267 Q9 0.710 -0.112 0.422 0.134 Q10 0.072 0.765 Q10 0.049 0.804 0.199 Q10 0.071 0.801 0.179 0.103 Subscale Subscale Subscale Alpha Alpha Alpha No Items No Items No Items 1 Q1,Q2,Q3,Q5,Q7,Q9 0.789 1 Q1,Q2,Q3,Q5,Q7,Q9 0.789 1 Q1,Q2,Q3,Q5,Q7,Q9 0.783 2 Q4,Q6,Q8,Q10 0.573 2 Q4,Q10 0.531 2 Q4,Q10 0.531 3 Q6,Q8 0.382 3 Q2,Q8 0.503 13 4 Q6 N/A
    14. 14. Summary & Questions Three studies involving 280 students assessing perceived usability of an LMS were conducted in various settings:  Education delivery method (blended, distance)  Time of evaluation (start, mid, end of semester) Results demonstrated the validity and reliability of the Greek SUS scale for LMS perceived usability evaluation First medium-large scale study (N=280) of Moodle’s perceived usability in Greece (and among the few worldwide)  Moodle SUS: 71.29 ± 1.51  Percentile rank: 60% (=> better than 60% of products in Sauro 2012 dataset of 446 studies and 5,000+ individual SUS responses) 14
    15. 15. Future Work & Preliminary results Effect of student’s characteristics (e.g. age, internet competence) on SUS scores (and score adjustments if required)  Age & SUS  rs = 0.13, p<0.05 (but 80% of sample aged 18 to 25)  r= –0.203, p<0.005 (Bangor et al, 2009)  Internet Competence & SUS  rs=0.25, p<0.01) (but only 8% of sample low competence) Effect of education delivery method on SUS scores (and score adjustments if required)  Education delivery method & SUS  Mann-Whitney U=5164, p=ns (but 85% of sample blended) More studies with different LMSs, course domain etc for 15 generalization of results
    16. 16. Extra Slides 16
    17. 17. References (1/2) A. Bangor, P. T. Kortum, and J. T. Miller, “Determining what individual SUS scores mean: adding an adjective rating scale,” Journal of Usability Studies, vol. 4, issue 3, 2009, pp.114–123. J. Brooke, “SUS: A quick and dirty usability scale,” in Usability evaluation in industry, P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I.L. McClelland, Eds. London, UK: Taylor and Francis, 1996, pp. 189– 194. N. Kostaras, and M. Xenos, "Assessing Educational Web-site Usability using Heuristic Evaluation Rules," Proc. PCI 2007 special session on Technological and Sociological Effects in the Use of e-Applications, Patras, 18-20 May, 2007, pp. 543-550. J. R. Lewis, “Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ,” SIGCHI Bulletin, vol. 23, issue 1, 1991, pp. 78–81. J. R. Lewis, “Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ,” Proc. of the Human Factors Society 36th Annual Meeting, Santa Monica, CA, 1992, pp. 1259-1263 J. R. Lewis, “IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use,” International Journal of Human-Computer Interaction, vol. 7, issue 1, 1995, pp. 57–78. J. R. Lewis, and J. Sauro, “The factor structure of the system usability scale”, Proc. Human Computer Interaction International Conference (HCII 2009), San Diego, CA, 2009, pp. 94–103 J. Nunnally, and I. Bernstein, Psychometric theory, McGraw-Hill Humanities/Social Sciences/Languages, 1994. J. Sauro & J. R. Lewis, (2012). Quantifying the User Experience: Practical Statistics for User Research (1st ed.). Morgan Kaufmann.
    18. 18. References (1/2) N. Tselios, N. Avouris, and V. Komis, “The effective combination of hybrid usability methods in evaluating educational applications of ICT: issues and challenges,” Education and Information Technologies Journal, vol. 13, issue 1, 2008, pp. 55–76. T. Tullis, and W. Albert, Measuring the user experience: collecting, analyzing, and presenting usability metrics, CA: Morgan Kaufmann Publishers Inc., 2008. T. Tullis, and J. Stetson, “A comparison of questionnaires for assessing website usability,” Proc. of the Usability Professionals Association (UPA) 2004 Conference, 2004, pp. 7–11.

    ×