Perceived usability is an important requirement of educational software, affecting greatly student’s learning effectiveness and overall learning experience. Although the System Usability Scale (SUS) questionnaire is considered as a de facto standard for perceived usability evaluation, surprisingly, few studies are reporting results from evaluations related to Learning Management Systems. In addition, given that the questionnaire comprises 10 questions presented in English, its applicability in studies involving speakers of other languages is questionable. In this paper, a first step towards standardization of a Greek version of SUS in the context of Learning Management Systems perceived usability evaluation is reported. To this end, three studies involving 280 university students in both blended and distance learning education were conducted. Analysis of the results demonstrated the validity and reliability of the Greek version of SUS.

1. Perceived usability evaluation of Learning
Management Systems: a first step towards
standardization of the System Usability Scale in Greek
Christos Katsanos* | ckatsanos@eap.gr
Nikolaos Tselios# | nitse@ece.upatras.gr
Michalis Xenos* | xenos@eap.gr
*Hellenic Open University
# Educational Science and Early Childhood Education, University of Patras
PCI 2012 | Piraeus, Greece | 5-7 Oct 2012

2. Perceived usability evaluation
of educational software
 Perceived usability is an important requirement of
educational software, affecting greatly student’s learning
effectiveness and overall learning experience (Kostaras &
Xenos 2011; Tselios et al. 2008)
 Numerous usability evaluation questionnaires designed to
assess the perceived usability of a product and/or software
are available at the moment (Brooke 1996; Lewis 1991, 1992,
1995; Tullis & Albert, 2008; Tullis & Stenson 2004)
Surprisingly, rather few studies report usage of questionnaires
to assess perceived usability of educational software 2

3. Perceived usability evaluation
of software: questionnaires
QUIS Questionnaire for User Interface Satisfaction Chin et al, 27 questions
1988
PUEU Perceived Usefulness and Ease of Use Davis, 1989 12 questions
NAU Nielsen's Attributes of Usability Nielsen, 1993 5 attributes
NHE Nielsen's Heuristic Evaluation Nielsen, 1993 10 heuristics
CSUQ Computer System Usability Questionnaire Lewis, 1995 19 questions
ASQ After Scenario Questionnaire Lewis, 1995 3 questions
PHUE Practical Heuristics for Usability Evaluation Perlman, 13 heuristics
1997
PUTQ Purdue Usability Testing Questionnaire Lin et al, 1997 100 questions
USE USE Questionnaire Lund, 2001 30 questions
SUS System Usability Scale Brooke, 10 questions
1996). 3

4. System Usability Scale (SUS)
 The most widely adopted for
perceived usability evaluation
of software/products (Tullis &
Albert, 2008; Sauro 2012)
 Why?
 Quick and dirty (10 questions)
 Standardized (valid, reliable)
 Summarized score (0 to 100)
 Reliable even with small
sample (10~12 users)
 SUS norms available
4
(Brooke, 1996)

5. Research Goals
 Examine the applicability of the SUS method in the context
of the usability evaluation of Learning Management
Systems (LMSs)
 Assess the validity and reliability of a Greek version of the
SUS questionnaire
 Non-native English speakers can confront problems in
understanding SUS compared to native speakers (Finstad 2006)
5

6. Methodology
 3 studies
 280 university students
 LMS: Moodle
 Courses on ICT & Education
 Both blended and distance education
 Various study times (sem. start, mid, end)
 SUS
 Between-subjects: Greek (N=221) vs English (N=59) versions
 Based on self-rated knowledge of English
 Greek version by 2 usability experts, native speakers
 Perceived usability 7-point rating [worst-imaginable, best-imaginable]
 Demographic questions (e.g. age, ICT skills, Moodle usage) 6

7. Studies Overview
Study Time N Gender Mean Delivery
Age method
1 Sem. End 45 45 Females 21.04 Blended
187 Females,
2 Sem. Start 191 20.45 Blended
4 Males
15 Females,
3 Sem. Mid 44 37.20 Elearning
29 Males
247 Females, Blended (2)
ALL {Sem. Start, Mid, End} 280 23.18
33 Males Elearning (1)
7

8. SUS English and Greek
Version
Item Greek Version of SUS English Version of SUS
Ννκίδω όηη ζα ήζεια λα ρξεζηκνπνηώ απηό ην I think that I would like to use this lesson's Moodle
Q1
ζύζηεκα Moodle ηνπ καζήκαηνο ζπρλά. system frequently.
Βξήθα απηό ην ζύζηεκα Moodle ηνπ καζήκαηνο I found the lesson's Moodle system unnecessarily
Q2
αδηθαηνιόγεηα πεξίπινθν. complex.
Σθέθηεθα όηη απηό ην ζύζηεκα Moodle ηνπ
Q3 I thought the lesson's Moodle system was easy to use.
καζήκαηνο ήηαλ εύθνιν ζηε ρξήζε.
Ννκίδω όηη ζα ρξεηαζηώ βνήζεηα από θάπνηνλ ηερληθό
I think that I would need the support of a technical
Q4 γηα λα είκαη ζε ζέζε λα ρξεζηκνπνηήζω απηό ην
person to be able to use this lesson's Moodle system.
ζύζηεκα Moodle ηνπ καζήκαηνο.
Βξήθα ηηο δηάθνξεο ιεηηνπξγίεο ζε απηό ην ζύζηεκα I found the various functions in this lesson's Moodle
Q5
Moodle ηνπ καζήκαηνο θαιά νινθιεξωκέλεο. system were well integrated.
Σθέθηεθα όηη ππήξρε κεγάιε αζπλέπεηα ζε απηό ην I thought there was too much inconsistency in this
Q6
ζύζηεκα Moodle ηνπ καζήκαηνο. lesson's Moodle system.
Φαληάδνκαη όηη νη πεξηζζόηεξνη άλζξωπνη ζα κάζνπλ
I would imagine that most people would learn to use
Q7 λα ρξεζηκνπνηνύλ απηό ην ζύζηεκα Moodle ηνπ
this lesson's Moodle system very quickly.
καζήκαηνο πνιύ γξήγνξα.
Βξήθα απηό ην ζύζηεκα Moodle ηνπ καζήκαηνο πνιύ I found the lesson's Moodle system very cumbersome
Q8
πεξίπινθν/δύζθνιν ζηε ρξήζε. to use.
Έληωζα πνιύ ζίγνπξνο/ε ρξεζηκνπνηώληαο απηό ην
Q9 I felt very confident using the lesson's Moodle system.
ζύζηεκα Moodle ηνπ καζήκαηνο.
Χξεηάζηεθε λα κάζω πνιιά πξάγκαηα πξηλ λα
I needed to learn a lot of things before I could get
Q10 κπνξέζω λα μεθηλήζω κε απηό ην ζύζηεκα Moodle
going with this lesson's Moodle system.
ηνπ καζήκαηνο.

9. Moodle-based course on ICT
& Education (screenshot)
9

10. Results (1/4) – Overview
Greek SUS English SUS
Item
Mean SD Mean SD
Q1 3.34 0.95 3.59 0.79
Mean SUS Mean Perceived
Students Q2 1.87 0.86 2.00 0.97
Study Score Usability Rating
(N)
(0-100) (1-7) Q3 3.75 0.89 3.74 0.91
1 45 74.50 ± 3.63 5.02 ± 0.20 Q4 1.47 0.77 1.66 0.90
2 191 70.48 ± 1.83 5.16 ± 0.11 3.69 0.83 3.47 0.80
Q5
3 44 71.48 ± 4.15 5.11 ± 0.27
Q6 1.89 1.04 2.14 0.90
All 280 71.29 ± 1.51 5.13 ± 0.09
Q7 3.74 0.88 3.80 1.03
Note: Numbers after the ± symbol are 95% confidence intervals for the
mean Q8 1.83 0.97 2.47 1.07
Q9 3.39 0.87 3.78 0.97
Q10 2.10 0.90 2.24 1.22
10

11. Results (2/4) – Validity Analysis
 Is the Greek SUS valid for LMS perceived usability evaluation?
 Validity = the extent to which an instrument, such as a questionnaire, measures what it is
intended to measure (Nunnally, 1994)
Concurrent Measure 1: Concurrent Measure 2:
Adjective Rating SUS English
rs=0.474, p<0.01 t(278)=1.149, p=0.252
11
*0.30 ~ 0.40 justify validity of questionnaires (Nunnally, 1994)

12. Results (3/4) – Reliability Analysis
 Does the Greek SUS measure LMS perceived usability reliably?
 Reliability = the extent to which an instrument, such as a questionnaire, yields the same
results under consistent conditions (Nunnally, 1994)
Σθέθηεθα όηη ππήξρε κεγάιε I thought there was too much
Q6 αζπλέπεηα ζε απηό ην ζύζηεκα inconsistency in this lesson's 12
Moodle ηνπ καζήκαηνο. Moodle system.

13. Results (4/4) – Factor Analysis
 Does the Greek SUS measure a single construct?
 English SUS was found to have two subscales: Learnability (Q4, Q10) and Usability (rest)
(Lewis & Sauro, 2009)
 Factor analyses using common verimax rotation => No reliable subscales
Component Component Component
Item Item Item
1 2 1 2 3 1 2 3 4
Q1 0.781 -0.096 Q1 0.782 -0.099 0.032 Q1 0.668 -0.167 0.393 -0.136
Q2 0.485 0.356 Q2 0.462 0.184 0.388 Q2 0.244 0.139 0.740 -0.004
Q3 0.652 0.339 Q3 0.645 0.431 0.033 Q3 0.668 0.381 0.160 0.005
Q4 0.124 0.648 Q4 0.108 0.736 0.099 Q4 0.143 0.726 0.115 0.036
Q5 0.642 0.293 Q5 0.626 0.208 0.259 Q5 0.670 0.173 0.136 0.304
Q6 0.032 0.537 Q6 -0.015 0.057 0.836 Q6 0.060 0.098 0.101 0.953
Q7 0.648 0.107 Q7 0.654 0.291 -0.168 Q7 0.752 0.241 -0.100 -0.046
Q8 0.280 0.598 Q8 0.244 0.341 0.567 Q8 0.056 0.321 0.719 0.197
Q9 0.812 0.078 Q9 0.799 -0.053 0.267 Q9 0.710 -0.112 0.422 0.134
Q10 0.072 0.765 Q10 0.049 0.804 0.199 Q10 0.071 0.801 0.179 0.103
Subscale Subscale Subscale
Alpha Alpha Alpha
No Items No Items No Items
1 Q1,Q2,Q3,Q5,Q7,Q9 0.789 1 Q1,Q2,Q3,Q5,Q7,Q9 0.789 1 Q1,Q2,Q3,Q5,Q7,Q9 0.783
2 Q4,Q6,Q8,Q10 0.573 2 Q4,Q10 0.531 2 Q4,Q10 0.531
3 Q6,Q8 0.382 3 Q2,Q8 0.503 13
4 Q6 N/A

14. Summary & Questions
 Three studies involving 280 students assessing perceived
usability of an LMS were conducted in various settings:
 Education delivery method (blended, distance)
 Time of evaluation (start, mid, end of semester)
 Results demonstrated the validity and reliability of the Greek
SUS scale for LMS perceived usability evaluation
 First medium-large scale study (N=280) of Moodle’s perceived
usability in Greece (and among the few worldwide)
 Moodle SUS: 71.29 ± 1.51
 Percentile rank: 60% (=> better than 60% of products in Sauro 2012
dataset of 446 studies and 5,000+ individual SUS responses) 14

15. Future Work & Preliminary
results
 Effect of student’s characteristics (e.g. age, internet competence) on
SUS scores (and score adjustments if required)
 Age & SUS
 rs = 0.13, p<0.05 (but 80% of sample aged 18 to 25)
 r= –0.203, p<0.005 (Bangor et al, 2009)
 Internet Competence & SUS
 rs=0.25, p<0.01) (but only 8% of sample low competence)
 Effect of education delivery method on SUS scores (and score
adjustments if required)
 Education delivery method & SUS
 Mann-Whitney U=5164, p=ns (but 85% of sample blended)
 More studies with different LMSs, course domain etc for 15
generalization of results

16. Extra Slides
16

17. References (1/2)
 A. Bangor, P. T. Kortum, and J. T. Miller, “Determining what individual SUS scores mean: adding an
adjective rating scale,” Journal of Usability Studies, vol. 4, issue 3, 2009, pp.114–123.
 J. Brooke, “SUS: A quick and dirty usability scale,” in Usability evaluation in industry, P. W. Jordan, B.
Thomas, B. A. Weerdmeester, and I.L. McClelland, Eds. London, UK: Taylor and Francis, 1996, pp. 189–
194.
 N. Kostaras, and M. Xenos, "Assessing Educational Web-site Usability using Heuristic Evaluation Rules,"
Proc. PCI 2007 special session on Technological and Sociological Effects in the Use of e-Applications,
Patras, 18-20 May, 2007, pp. 543-550.
 J. R. Lewis, “Psychometric evaluation of an after-scenario questionnaire for computer usability studies:
The ASQ,” SIGCHI Bulletin, vol. 23, issue 1, 1991, pp. 78–81.
 J. R. Lewis, “Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ,”
Proc. of the Human Factors Society 36th Annual Meeting, Santa Monica, CA, 1992, pp. 1259-1263
 J. R. Lewis, “IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions
for use,” International Journal of Human-Computer Interaction, vol. 7, issue 1, 1995, pp. 57–78.
 J. R. Lewis, and J. Sauro, “The factor structure of the system usability scale”, Proc. Human Computer
Interaction International Conference (HCII 2009), San Diego, CA, 2009, pp. 94–103
 J. Nunnally, and I. Bernstein, Psychometric theory, McGraw-Hill Humanities/Social Sciences/Languages,
1994.
 J. Sauro & J. R. Lewis, (2012). Quantifying the User Experience: Practical Statistics for User Research
(1st ed.). Morgan Kaufmann.

18. References (1/2)
 N. Tselios, N. Avouris, and V. Komis, “The effective combination of hybrid usability methods in evaluating
educational applications of ICT: issues and challenges,” Education and Information Technologies Journal,
vol. 13, issue 1, 2008, pp. 55–76.
 T. Tullis, and W. Albert, Measuring the user experience: collecting, analyzing, and presenting usability
metrics, CA: Morgan Kaufmann Publishers Inc., 2008.
 T. Tullis, and J. Stetson, “A comparison of questionnaires for assessing website usability,” Proc. of the
Usability Professionals Association (UPA) 2004 Conference, 2004, pp. 7–11.