Paper presented at the 22nd International Conference on Human-Computer Interaction (HCII 2020). Evaluation of the Virtual Mobility Learning Hub Diana Andone, Silviu Vert, Vlad Mihaescu, Daniela Stoica, Andrei Ternauciuc Politehnica University of Timisoara, Romania

1. Diana Andone, Silviu Vert, Vlad Mihăescu,
Daniela Stoica, Andrei Ternauciuc
Evaluation of the Virtual
Mobility Learning Hub
Politehnica University of Timisoara, Romania
diana.andone@upt.ro, silviu.vert@upt.ro, vlad.mihaescu@upt.ro,
andrei.ternauciuc@upt.ro, daniela.stoica@student.upt.ro
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the
European Union under grant no. 2017-1-DE01-KA203-003494. Neither the European Commission nor the project's
national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

2. Open Virtual Mobility
Learning Hub
https://hub.openvirtualmobility.eu/
▷ Innovative multilingual (7 languages) ICT-
based environment (as a directory of virtual
mobility attributes).
▷ Promotes collaborative learning,
connectivism social networking as an
instructional method, open educational
resources OERs as the main content, open
digital credentials as recognition and
validation of VM skills which can be applied
to all ages and levels of digital education.
▷ For students living in different parts of the
world to learn and collaborate and to have
free access to some relevant Massive Open
Online Courses (MOOCs).
2

3. 139 participants
5 usability testing methods
8 months period of testing
3

4. Related work
Adaptable usability
checklist for evaluating the
user interface of MOOCs
Johansson, S., Frolov, I.: An
adaptable usability checklist for
MOOCs: a usability evaluation
instrument for massive open
online courses, Informatik
Student Paper Master (INFSPM)
(2014)
Methodology for assessing
user satisfaction of MOOCs
Korableva, O., Durand, T.,
Kalimullina, O., Stepanova, I.:
Usability testing of MOOC:
identifying user interface
problems. In: Proceedings of the
21st International Conference
on Enterprise Information
Systems, ICEIS 2019, pp. 468–
475. SciTePress (2019)
Comparative usability
evaluation of edX,
Coursera and Udacity
Tsironis, A., Katsanos, C., Xenos,
M.: Comparative usability
evaluation of three popular
MOOC platforms. In: 2016 IEEE
Global Engineering Education
Conference (EDUCON), pp.
608–612. IEEE (2016)
4
Systematic review of 23 usability and user
experience evaluations of LMS
Salas, J., et al.: Guidelines to evaluate the usability
and user experience of learning support platforms:
a systematic review. In: Ruiz, P.H., Agredo-
Delgado, V. (eds.) HCI-COLLAB 2019. CCIS, vol.
1114, pp. 238–254. Springer, Cham (2019).
https://doi.org/10.1007/978-3-030-37386-3_18
Usability evaluation of the mobile display of
an LMS platform
Ivanc, D., Vasiu, R., Onita, M.: Usability evaluation of
a LMS mobile web interface. In: Skersys, T., Butleris,
R., Butkiene, R. (eds.) ICIST 2012. CCIS, vol. 319, pp.
348–361. Springer, Heidelberg (2012).
https://doi.org/10.1007/978-3-642-33308-8_29
MOOCLMS

5. Research questions:
Q1. Can a Moodle LMS sustain
fully open, online, not tutorized
courses?
Q2. What are the experiences
that real students might have as
learners in the VMLH?
Q3. Are the OpenVM MOOCs
error-free and ready to become
available for the HEI market?
Methodology (1)
Testing methods:
1. Focus Group – Q1, Q2
2. User Observation – Q2, Q3
3. Error Testing – Q3
4. Survey –Q1, Q2, Q3
5. Expert Review–Q1, Q3
5

6. Participants:
▷ 21 second year students from the
Multimedia Technologies Master
▷ 17 second year students from the
Communication, Public Relations
and Digital Media Master
▷ 98 first year students from
various technical Master
Programs
▷ 3 eLearning Experts (university
professors)
Methodology (2)
Tested MOOCs:
7 mini MOOCs composed
of 3 courses each
(Foundation, Intermediate
and Advanced), in total 21
courses
Period:
8 months (April 2019 until
November 2019)
6

7. Focus Group
7
Multimedia Center,
Politehnica University of
Timisoara
21 master students, 23-26
y/o, employed in IT
1 day
tasks, discussion

8. User Observation
8
Multimedia Center,
Politehnica University of
Timisoara
5 master students, 23-26
y/o, employed in IT
1 day
video recording &
confidentiality agreement,
post questionnaire, tasks,
guidelines, etc

9. Error testing
9
online
98 master students, 22-25
y/o, employed in IT
2 months
online collaborative
document for reporting the
errors

10. Questionnaire
10
online
17 master students, 22-24
y/o, moderate IT skills
2 weeks
online form integrated into
the university LMS

11. Expert evaluation
11
Multimedia Center,
Politehnica University of
Timisoara
3 eLearning experts
(university teachers)
3 days
discussion

12. Examples of usability problems &
recommendations
Usability problem Severity rating
The header on the page was too big 1 Cosmetic
Not everything is translated into English. 2 Minor
The tasks and activities weren’t displayed correctly and the students
did not received the badges after every course. 3 Major
12
The participants believe that for some of the courses, the structure should
be modified in order to be more user friendly because they had some issues
understanding exactly what they are supposed to do.

13. Conclusions
A3
• The MOOCs are
not error-free and
improvements
need to be done
before being made
available to the
HEI market.
• Most of the
improvements are
in the area of the
content of the
courses.
A1
• A Moodle-based
LMS can sustain
fully open, online,
not tutorized
courses.
• Is dependent on
actual course
content and
teacher setup of
the learning
environment.
A2
• Good experience
of students in the
VMLH.
• Rated better than
the faculty
courses with good
quality activities.
• However, students
often stumbled
upon small to
medium
annoyances.
13

14. “The major contribution of the paper is in using a mix of 5 usability
evaluation methods, with a large group of participants (139 persons), in an
extended period of time (8 months), to derive usability issues pertaining to a
MOOC platform.
Also, the study shows how this mixed usability testing can be done in a
university environment, with a practical outcome but also with a pedagogical
purpose.
14

15. The improved version of the learning hub is now
used by students and professors in 7 universities.
15
Universitat de
les Illes
Balears
Beuth Hochschule
für Technik
Universita Degli
Studi Roma Tre
Politehnica University
of Timişoara
Katholieke Universiteit
Leuven
Open Universiteit
Nederland
Fondation
UNIT/AUNEGE

16. Thanks!
Questions are welcome
You can find us at:
diana.andone@upt.ro, silviu.vert@upt.ro, vlad.mihaescu@upt.ro,
andrei.ternauciuc@upt.ro, daniela.stoica@student.upt.ro
e-Learning Center
Politehnica University of Timisoara
https://elearning.upt.ro/ro/
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the
European Union under grant no. 2017-1-DE01-KA203-003494. Neither the European Commission nor the project's
national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.
Presentation
template by
SlidesCarnival