Presentation by Francesca Amenduni, Ph.D. Student, University of Roma Tre, EDEN NAP member, at the 2019 European Distance Learning Week's third-day webinar on "Future perspective in open routes: the quality and assessment dimension?" - 13 November 2019 Recording of the discussion is available: https://eden-online.adobeconnect.com/p8i31ynv7v28/ & https://www.youtube.com/watch?v=7yb8ekFP3ps

1. Assessing quality in an Open Virtual Mobility
MOOC
Preliminary results from the first pilot phase cycle
Francesca Amenduni

2. Assessing quality in an Open
Virtual Mobility MOOC
Preliminary results from the first pilot phase
cycle
Francesca Amenduni
PhD student at Roma Tre University
francesca.amenduni@uniroma3.it
Twitter: @amendoonia

3. The Project Open Virtual
Mobility Erasmus +
Open Virtual Mobility (openVM) is a European strategic partnership dedicated to creating
accessible opportunities to achieve virtual mobility skills and to ensure higher uptake of virtual
mobility in higher education in Europe. It is funded under the European Erasmus+ Program.
The group of the University of Roma
Tre coordinated by Antonella Poce
is responsible for the Output 6
named "OER, MOOC, and Pilots".
This output is aimed at designing
VM OERs and the VM MOOC and
ensuring the project sustainability
through a piloting phase.

4. Pilot phase - methodology
Our pilot-phase was realised in order to assess the quality of
the OpenVM MOOCs and its main components:
- Matching tool and group formation (Output 3);
- E-assessment, more specifically quizzes, e-portfolio and
peer-assessment (Output 4);
- Gamification and badges (Output 5);
- Learning materials and course structures (Output 6),
The pilot phase is organized in three iterations designed
according to the principles of the DBR and ADDIE (Output
7): 1. a pre-pilot phase (Dec. 2018- Jan. 2019) 2. the first
pilot-phase cycle (Sept- - Dec. 2019) and 3. the second-pilot
phase cycle (Jan. - May 2020).

5. The MOOC assessment questionnaire
https://docs.google.com/forms/d/1OJD62k4IveuUbDeFo-
sw_sO4YU52gsoFyWSf1GsIkK0/edit#responses
The questionnaire is organized in the following sections:
1. Personal details (such as age, gender, affiliation and role);
2. General questions regarding the MOOC.
3. Questions regarding badges and gamification.
4. Questions regarding technical aspects.
5. Questions regarding foundation, intermediate and advanced
level;
6. The last section was aimed to investigate to which extent the
MOOC support SRL processes. In this section participants have
to answer both to closed and open-ended questions.
*For sections from 1 to 6 participants are required to express their
level of agreement with statements regarding the MOOC on a Likert
scale from 1 (strongly disagree) to 5 (totally agree);

6. Preliminary results of the pilot phase
The miniMOOC “Active Self-regulated Learning” was attended by the 47,6% of the respondents. The second
miniMOOC most attended was the “Media and Digital Literacy” with 23,6% of the answers, followed by
“Intercultural skills”, “Open-mindedness”, “Open Education and Virtual Mobility”, “Networked Learning”,
“Autonomy-drive learning” and “Collaborative learning”.
Number of
respondents: 250* the
number is constantly
growing.
Average year: 24
years
Target group: 99%
students; 1% teachers.

7. Despite most of the participants
come from partners’ universities, a
small percentage of external
institutions took part in the MOOCs.

8. Preliminary results of the pilot phase
Foundation level is the only level that not
requires tutoring! In the intermediate level it is
necessary to manually create groups and in the
advanced level to manually activate peer-
evaluation tasks.

9. Preliminary results of the pilot phase
Regarding technical aspects of the MOOC we see that participants express general positive evaluation. In
particular, the length of the video was considered good (Average = 3,86), the sound of the video was clear
(Average = 3,81) and the technological environment is friendly and usable (Average = 3,82). A less positive
agreement regards the statements the MOOC layout positively affects my learning experience and the system is
easy to use with average lower than 3,65.

10. Preliminary results of the pilot phase
Participants tend to disagree with negative statements regarding the MOOCs with average lower than 3 (on the
bottom of the graph). On the other hand, participants agree with positive statements. In particular, participants
seem to appreciate the use of different kinds of contents - OERs (Average = 4).

11. Preliminary results of the pilot phase
Finally, also for badges and gamification participants express general positive evaluation, in particular toward
the badge design (Average = 3,65).

12. Next phases
The next phase are:
- Since the sample will be enough consistent, we will carry out more
elaborated statistical analysis after the end of the first pilot cycle in
December. For example: 1. Are there any statistical differences among
scores for each MOOC? 2. Are there are statistical differences among
scores within a MOOC for each level?
- Collecting feedback to implement the MOOC before the the second
pilot cycle (February 2020);
- Carrying out the second pilot cycle analysing the results.

13. Thank you for your attention.
Visit our learning hub
https://hub.openvirtualmobility.eu
Follow the project on twitter @openVM_erasmus
For more information regarding our pilot phase and
suggestions to improve the MOOC, you can write to
Antonella Poce and me
Antonella.poce@uniroma3.it
Francesca.amenduni@uniroma3.it