Presentation by Antonella Poce and Francesca Amenduni during Webinar EADTU & Open VM on Friday 19 June 2020

1. A Massive Open Online Course designed to support the
development of Virtual Mobility Transversal Skills:
preliminary evaluation results from European participants
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.
Antonella Poce,
Francesca Amenduni, Carlo De Medio, Maria
Rosaria Re, Mara Valente, Francesco Agrusti.

2. Background
• Despite the growing acknowledged of Virtual Mobility, only a few researches have
investigated the impact of Virtual Mobility initiatives on participants, and most of
them includes small scale studies (Hilliard, 2004; Frydenberg, & Andone, 2010;
Costa, & Balula, 2014);
• Results of the previous experience suggest the critical role of participants’
transversal skills in Virtual Mobility Experience (Poulová, Černá &Svobodová,
2009);
• MOOCs are now being considered and applied by many institutions around the
world as a valid internationalization instrument (Knight, 2014). However, Amirault
and Visser (2010) show that virtual program offerings do not automatically cross
borders, nor result in the same effects everywhere.
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

3. The goal of the research
The OpenVM MOOC design was developed by all the partners through the
coordination of the Roma Tre Team. The OpenVM is aimed at developing those
transverse skills necessary to be successfully involved in a Virtual Mobility programme
in an international digital learning environment.
Our preliminary research questions are:
• To which extent students enjoyed the OpenVM MOOC design and its main
components?
• Are there any differences in the assessment of the 8 miniMOOCs and subMOOCs?
• Does the MOOC support students’ self-regulated learning?
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

4. Pilot phase - methodology
Our pilot-phase was carried out in order to assess the quality of the OpenVM
MOOCs and its main components:
- Matching tool and group formation (Output 3);
- E-assessment, more specifically quizzes, e-portfolio and peer-assessment
(Output 4);
- Gamification and badges (Output 5);
- Learning materials and course structures (Output 6),
The pilot phase is organized in three iterations designed according to the principles
of the DBR and ADDIE (Output 7): 1. a pre-pilot phase (december 2018- january
2019) 2. the first pilot-phase cycle (september - december 2019) and 3. the second-
pilot phase cycle (2020).
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

5. The MOOC assessment questionnaire
https://docs.google.com/forms/d/1OJD62k4IveuUbDeFo-sw_sO4YU52gsoFyWSf1GsIkK0/edit#responses
Participants were required to express their level of agreement with a set of statements related to specific MOOC design elements on a
Likert scale from 1 (strongly disagree) to 5 (totally agree).
• Personal details: age, gender, affiliation and role. In this section participants are required to say which of the eight mini-MOOCs they
are assessing;
• Questions regarding the overall MOOC design: learning experience, quality of content instruction and support for learning;
• Questions regarding digital credentials and meaningful gamification: quality of design, motivation, engagement and possibilities of
choice;
• Questions regarding technical aspects: use and usability;
• Questions regarding the foundation level of a mini-MOOC: duration, language, content, use of multimedia;
• Questions regarding the intermediate level of a mini-MOOC: extending questions from the foundation level by questions related to
the matching tool and group formation activity, which are specific design elements used at this level;
• Questions regarding the advanced level of a mini-MOOC: extending questions from the foundation and intermediate levels by
questions related to e-portfolio and peer-assessment activities, which are specific design elements at this level;
• Questions related to the investigation about the extent to which MOOCs supported self-regulated learning. In this section
participants are required to answer also to open-ended questions.

6. Sample
Characteristics – first pilot cycle
716 (F = 498; M = 215; Not
specified = 3) participants took
part in the pilot-phase.
Most of the participants were
university students and only 14
teachers participated in the
survey.
Gender Number %
• Female 498 69,6%
• Male 215 30%
• Prefer Not to say 3 4%
Role
• University student 699 97,6%
• Teacher 14 2%
• Other 3 0,4%
Age
• Less than 20 153 21,4
• Between 21 and 23 348 48,6
• Between 24 and 26 100 14,0
• Between 27 and 30 44 6,1
• Between 31 and 35 34 4,7
• More than 36 37 5,2
Total 716 100%
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

7. MOOC attendance
27% took part and
assessed the MOOC Active Self-
regulated Learning
22% Media and Digital Literacy
15% “Open-mindedness”
14%“Intercultural Skills”
7% “Networked Learning”
6%“Autonomy-drive learning”
5%“Collaborative Learning” and
“Open Education and Virtual
Mobility”
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

8. Affiliation
Roma Tre
University
Beuth University
of Applied
Sciences
Universitatea
Politehnica
Timisoara
Universitat de
les Illes
Balears (UIB)
AUNEGE Open
Universiteit –
Welten
Instituut
External
institutions
Total
MOOC Media and Digital Literacy 0 74 61 1 11 1 7 155
Intercultural Skills 6 1 76 1 3 2 13 102
Active Self-Regulated
Learning
132 24 17 21 0 0 0 194
Autonomy drive-learning 4 0 34 1 0 1 0 40
Networked learning 1 6 40 0 0 0 3 50
Collaborative learning 2 1 36 0 0 0 0 39
Open-mindedness 5 2 100 1 1 0 0 109
Open Education and Virtual
Mobility
8 9 4 4 0 0 2 27
Total 158 117 368 29 15 4 25 716
MOOCs most attended by each partner institution
Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494.
Neither the European Commission nor the project's national funding agency DAAD are responsible for the content or liable for any losses or damage resulting of the use of
these resources.

9. General evaluation of the MOOCs -
Positive statements
Average N SD
Networked learning 3,5945 50 1,02593
Media and Digital Literacy 3,6938 155 0,65947
Collaborative learning 3,7203 39 0,72619
Active Self-Regulated Learning 3,7381 194 0,6769
Open-Education and Virtual
Mobility 3,7542 27 0,61802
Average 3,7689 716 0,72937
Open-mindedness 3,8182 109 0,769
Autonomy driven-learning 3,8477 40 0,72797
Intercultural Skills 3,9661 102 0,71558
The general evaluation of the MOOCs was
quite positive.
The average score for each MOOC is always
higher than 3,5 out of 5 points
(we used a Likert Scale from 1 to 5, where
the median is 3).
The total average was 3,77.
Three MOOCs out of eight obtained scores
higher than the average:
1. Open-mindedness (109 answers)
2. Autonomy-driven learning (40 answers)
3. Intercultural skills (102 answers).

10. Badge evaluation - Positive
statements
Average N SD
Active Self-Regulated Learning 3,5351 194 0,78093
Networked learning 3,56 50 1,06904
Media and Digital Literacy 3,591 155 0,84741
Intercultural Skills 3,6922 102 0,76853
Open-mindedness 3,7193 109 0,84989
Collaborative learning 3,7385 39 0,83874
Autonomy driven-learning 3,93 40 0,72402
The general evaluation of the badges was quite
positive.
The average score for badge is always higher
than 3,5 out of 5 points
(we used a Likert Scale from 1 to 5, where the
median is 3).
Please consider that we do not have badges for
the MOOC “Open-Education and Virtual
Mobility”.
Participants expressed a preference for
badges in three MOOCs:
- Open-mindedness
- Collaborative learning
- Autonomy driven-learning

11. Gamification features
evaluation - Positive
statements
Average N SD
Networked learning 3,18 50 0,94868
Collaborative learning 3,2487 39 0,69995
Media and Digital Literacy 3,2613 155 0,66189
Active Self-Regulated Learning 3,2964 194 0,62216
Open Education and Virtual
Mobility 3,3148 27 0,6125
Average 3,3226 716 0,68037
Open-mindedness 3,3917 109 0,69949
Autonomy driven-learning 3,41 40 0,68605
Intercultural Skills 3,4578 102 0,63315
The general evaluation of the gamification
features was still more than 3 but lower than
3,5.
Please consider that we do not have badges
for the MOOC “Open-Education and Virtual
Mobility”.
The total average was 3,32
Participants expressed a preference for
gamification in three MOOCs:
- Open-mindedness
- Autonomy driven-learning
- Intercultural skills

12. Technical features evaluation -
Positive statements
Averag
e N SD
Open Education and Virtual Mobility 3,5704 27 0,66957
Networked learning 3,6449 49 1,09412
Media and Digital Literacy 3,6768 151 0,8339
Active Self-Regulated Learning 3,7403 191 0,71642
Average 3,7946 704 0,8078
Collaborative learning 3,8378 37 0,78717
Autonomy driven-learning 3,88 40 0,77499
Open-mindedness 3,9 108 0,83184
Intercultural Skills 4,0436 101 0,74222
The technical features evaluation of the MOOCs was
quite positive.
The average score for each MOOC is always higher
than 3,5 out of 5 points
(we used a Likert Scale from 1 to 5, where the
median is 3).
The total average was 3,79.
Four MOOCs out of eight obtained scores higher
than the average:
1. Collaborative learning
2. Autonomy-driven learning
3. Open-mindedness
4. Intercultural skills

13. Conclusive remarks on the first part of the
questionnaire
From a general overview we
can conclude that four
MOOCs at the moment were
the most appreciated
considering
1. general features
2. badges
3. gamification
4. technical features
1. Collaborative
learning
2. Autonomy-driven
learning
3. Open-mindedness
4. Intercultural skills

14. 92% of participants took part in the foundation level, 51,6% in the
intermediate level and 30,6% in the advanced level in one of the 8
miniMOOCs.
Which level did participants attend?

15. This picture shows a comparison of the average
answers given for foundation, intermediate and
advanced level.
For each statement, in all the levels average
scores are higher than 3,55.
We can see that course duration is considered
better in foundation level than intermediate
and advanced level.
Usually foundation level achieved higher
average scores.
However, multimedia contents, and language
are more appreciated in advanced level.

16. E-portfolio and peer assessment
activity
Average N SD
Networked learning 3 17 1,1607
Media and Digital Literacy 3,4943 58 0,84982
Active Self-Regulated Learning 3,6 30 0,74997
Open-mindedness 3,6471 51 1,05496
Collaborative learning 3,6923 13 0,93751
Autonomy drive-learning 3,9167 12 0,76706
Intercultural Skills 3,9394 33 0,85576
Group formation and matching
tool
Average N SD
Networked learning 3,5486 24 1,39658
Media and Digital Literacy 3,9627 85 0,99116
Active Self-Regulated Learning 3,966 54 1,03926
Collaborative learning 4,1587 21 1,07945
Open-mindedness 4,1689 73 1,11958
Intercultural Skills 4,2091 55 1,22484
Autonomy drive-learning 4,2826 23 0,93802
The general evaluation of
- e-portfolio and peer-assessment
- group formation and matching tool
was quite positive.
The average score for Group Formation is always
higher than 3,5 out of 5 points;
The average score for E-portfolio and peer-
assessment is lower than 3,5 only for
1. Networked Learning
2. Media and Digital Literacy.
Participants expressed a preference for e-
assessment and group formation in:
- Open-mindedness
- Collaborative learning
- Autonomy drive-learning

17. Qualitative research - first pilot
(students opinion)
Examples of qualitative answers analysed through
thematic analysis (N=20)
I enjoyed doing the Course
need more explanation
The Foundation Level MOOC was pretty short and that
took an influence on my answers
it helped me a little bit
self observation
It helped me to use new Technologies to learn and to
exercise my brain
Thematic analysis
Helpful to improve strategy of study: 14
SRL skills development: 10
General Enjoyment: 9
Generally Useful/Helpful: 9
Refers to technologies: 5
Negative comments: 3
Not categorized: 2

18. Sustainability - follow up suggestions
Create resources for teachers on how to integrate the OpenVM MOOCs in their courses
Different OERs for teachers and students could be created taking into account:
- How to create groups (for teachers)
- How to work a good e-portfolio out (for teachers and students)
- How to support students’ peer assessment activities (both for teachers and
students)
These OERs should be based on the results of pilot experience.

19. References
• Amirault, R. J., & Visser, Y. L. (2010). The impact of e-learning programs on the internationalization of the university. New York, NY: Nova Science Publishers.
• Bates, A. (2015). Variations in MOOC designs. BC Open Texbooks.
• Buchem, I., Tur, G. & Urbina, S. (2018). Quality assurance for attainment, assessment and recognition of virtual mobility skills in context of open education. QA Framework in the Open Virtual Mobility
project. Paper presented at Edulearn Conference 2-4 July 2018. DOI: 10.21125/edulearn.2018.0318 Retrieved from https://iated.org/concrete3/view_abstract.php?paper_id=65036
• Costa, N., & Balula, A. (2014). Virtual mobility and learning for PhD students of six European countries-students' programme evaluation. Profesinis rengimas: tyrimai ir realijos, 2014, nr. 25, p. 178-188.
• Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. Journal of interactive Media in education, 2012(3). DOI: http://doi.org/10.5334/2012-18
• Frydenberg, M., & Andone, D. (2010). Two Screens and an Ocean: Collaborating across Continents and Cultures with Web-Based Tools. Information Systems Education Journal, 8(55), n55.
• Hilliard, A. (2004, January). Outline and Evaluation of a Joint European and Canadian Virtual Mobility: e-Learning Project. In ECEL2006-5th European Conference on e-Learning: ECEL2006(p. 163). Academic
Conferences Limited.
• Knight, J. (2014). Three generations of crossborder higher education: New developments, issues and
• Konert, J., Burlak, D., & Steinmetz, R. (2014, September). The group formation problem: an algorithmic approach to learning group formation. In European Conference on Technology Enhanced
Learning (pp. 221-234). Springer, Cham.
• Poulová, P. E. T. R. A., Černá, M. I. L. O. S. L. A. V. A., & Svobodová, L. (2009). University Network? Efficiency of Virtual Mobility. In Proceedings of the 5th WSEAS/IASME International Conference on
EDUCATIONAL TECHNOLOFIES (EDUTE09) (pp. 87-92).
• Rajagopal, K., & Firssova, O. (2018, April). Virtual Mobility in the context of Open Education: re-establishing the boundaries. In Open Education Global Conference 2018: Transforming education through
open approaches. Routledge handbook of international education and development (pp. 340-358). Abingdon: Routledge.
• Teichler, U. (2004). The changing debate on internationalization of higher education. Higher Education, 48, 5-26.
• Teresevičienė M., Volungevičienė A., and Daukšienė E. (Eds.) (2011). Virtual mobillity for Teachers and Students in Higher Education. Kaunas: Vytautas Magnus University.
• Yuan, L., & Powell, S. J. (2013). MOOCs and open education: Implications for higher education. Retrieved from: https://e-space.mmu.ac.uk/619735/1/MOOCs-and-Open-Education.pdf
• Zimmerman, B. J. (2013). Theories of self-regulated learning and academic achievement: An overview and analysis. In Self-regulated learning and academic achievement (pp. 10-45). Routledge.