Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
MOOQ and the
Quality of MOOCs:
Findings & Tools
Christian M. Stracke & Esther Tan
Open University of the Netherlands
Global cooperation: ECNU & KNOU
Global initiative ICORE for OR & OE
International WLS /...
Open University of the Netherlands
The Quality of MOOCs
Let’s Learn to Learn
Seamless Learning
Dr. Esther Tan
Technology-E...
Presentation Outline
1. The Quality of MOOCs: What is the problem?
2. A Conceptual Framework towards MOOC QRF:
Theoretical...
Any discussion on the quality of MOOCs should consider the goals
of both, the MOOC learner and the provider. (Hayes, 2015)...
A Research Framework for MOOQ
“When one designs any course, one has to have some learner
cohort in mind.” (Macleod et al.,...
Global MOOC
Quality Survey
Quality Reference Framework with
criteria & checklist for MOOC design
Our main goal is the collaboration with all
to impro...
 Subjects of Investigation:
Learners, Designers, Facilitators & Providers
 Mixed Methods Approach:
Quantitative & qualit...
Instruments of Measure
MOOC
Learners
MOOC
Designers
MOOC
Facilitators
MOOC
Providers
TOTAL
Global
MOOC
Quality
Survey
166
...
MOOC Survey Constructs
Constructs Learners Designers Facilitators
Experience with MOOC X X X
Learning Objectives X X X
Dur...
Global MOOC
Quality Survey
(GMQS)
Global MOOC Quality Survey
Demographic Profile
Age range of all survey participants (learners, designers & facilitators) by gender
Demographic Profile
Educational level of all survey participants (learners, designers & facilitators) by gender
n VB B N G VG
Learning
experience
166 4 4 13 75 70
Learning Experience (Learners)
n VB B N G VG
Design
experience
68 1 2 13 33 19
Design Experience (Designers)
Interaction from Learners‘ Perspective
n N/A SD D N A SA
LF 146 20 5 13 48 37 23
LL 146 15 3 17 34 51 26
LR 146 9 2 8 25 6...
n R2 M2 p
LF by
learners
125 .094 9.382 .000***
LL by
learners
130 .101 10.818 .000***
LR by
learners
136 .112 12.286 .000...
Interaction from Designers‘ Perspective
n N/A SD D N A SA
LF 52 2 1 5 11 24 9
LL 52 1 1 3 11 19 17
LR 52 3 1 0 4 22 22
GG ...
n R2 M2 p
LF by
designers
49 .003 0.109 .703
LL by
designers
50 .043 1.595 .143
LR by
designers
48 .046 1.537 .138
GG by
d...
Interaction from Facilitators‘ Perspective
n N/A SD D N A SA
LF 32 0 1 3 3 15 10
LL 32 1 0 3 2 11 15
LR 32 0 0 0 5 12 15
G...
n R2 M2 p
LF by fa-
cilitators
30 .181 0.242 .015*
LL by fa-
cilitators
29 .021 0.296 .435
LR by fa-
cilitators
30 .030 0....
Open-ended
Questions
(Learners)
No of respondents across the six domains
Domain Learners Designers Facilitators
Social Sciences, Humanities &
Law
24 9 2
E...
MOOC Open-ended Questions (OEQ)
MOOC Learners
Q1. What were the three main strengths of the MOOC?
Q2. What were the three ...
Age Range of MOOC Learners
18-24 25-34 35-44 45-54 55-64 65-74 >75 Total
Business,
Management &
Economics
0 3 6 9 1 1 0 20...
Educational Level of Learners
No
School-
ing
High
school
Bachelor’s
degree
Masters
degree
Doctorate
degree Total
Business,...
Gender of MOOC Learners
Male Female Other Total
Business,
Management &
Economics
12 8 0 20
Nature,
Environment &
Health
14...
Learning Experience
Very
Bad Bad Neutral Good Very
Good Total
Business,
Management &
Economics
1 0 2 9 8 20
Nature,
Enviro...
Open-ended Questions (MOOC Learners)
% of Activities Completed
None <25% =50% >75% >100% Total
Business, Management &
Econ...
Open-ended Questions (MOOC Learners)
Q1. What were the three main strengths of the MOOC?
0
10
20
30
40
50
60
70
80
90
Curr...
Open-ended Questions (MOOC Learners)
Q2. What were the three main weaknesses of the MOOC?
0
5
10
15
20
25
30
35
Curriculum...
Open-ended Questions (MOOC Learners)
Three main strengths and weaknesses of the MOOC
Strengths Weaknesses
Curriculum Desig...
Open-ended
Questions
(Designers)
MOOC Open-ended Questions (OEQ)
MOOC Designers
Q1. Which were the main design decisions that you made
during the developme...
Age Range of MOOC Designers
25-34 35-44 45-54 55-64 65-74 >75 Total
Business, Management
& Economics
1 1 3 0 0 0 5
Nature,...
Educational Level of Designers
High
school
Bachelor’s
degree
Masters
degree
Doctorate
degree Total
Business, Management
& ...
Gender of MOOC Designers
Male Female Total
Business, Management
& Economics
3 2 5
Nature, Environment &
Health
2 4 6
Scien...
Open-ended Questions (MOOC Designers)
No. of MOOCs Designed
1 2 - 4 5 - 9 >10 Total
Business, Management &
Economics
3 2 0...
Design Experience
Very
Bad Bad Neutral Good Very
Good Total
Business,
Management &
Economics
0 0 0 5 0 5
Nature,
Environme...
Open-ended Questions (MOOC Designers)
Q1. Which were the main design decisions that you made during the
development of the...
Open-ended Questions (MOOC Designers)
Q2. Which were the three biggest difficulties that you faced when designing the
MOOC...
Open-ended Questions (MOOC Designers)
Successful Decisions Biggest Challenges
Curriculum Design & Delivery
• Content deliv...
Open-ended
Questions
(Facilitators)
MOOC Open-ended Questions (OEQ)
MOOC Facilitators
Q1. Which were the main decisions that you made during the
facilitation ...
Age Range of MOOC Facilitators
25-34 35-44 45-54 55-64 65-74 >75 Total
Business, Management
& Economics
0 1 0 0 0 2 3
Natu...
Educational Level of MOOC Facilitators
High
school
Bachelor’s
degree
Masters
degree
Doctorate
degree Total
Business, Manag...
Gender of MOOC Facilitators
Male Female Total
Business, Management
& Economics
0 3 3
Nature, Environment &
Health
1 0 1
Sc...
Open-ended Questions (MOOC Facilitators)
No. of MOOCs Facilitated (OEQ)
1 2 - 4 5 - 9 >10 Total
Business, Management &
Eco...
Facilitation Experience
Good Very Good Total
Business, Management
& Economics
3 0 3
Nature, Environment &
Health
0 1 1
Sci...
0
1
2
3
4
5
6
7
8
9
Provide
guidelines on
attendance, quiz
& test
Monitor peer
review process
& forum
discussion
Show pres...
0
1
2
3
4
5
Difficult to
engage &
retain learners
Integrate IT
& Media to
support
learning
Lack IT
competencies
(learners)...
Semi-Structured
Interviews Questions
(Providers)
Categories Descriptor Indicators
Role of MOOCs in HE Providers perspective
on the role of MOOCs
in the current HE
national...
Categories Descriptor Indicators
Partnerships Establishment, organization
and management of
partnerships
 Partnership str...
Categories Descriptor Indicators
Institutional
organizational
strategies
Description of the MOOC
organization and
developm...
Categories Descriptor Indicators
Sustainability Characterization of the
return on investment
 Funding source(s):
1. Busin...
Categories Descriptor Indicators
Evaluation Design of the evaluation
process for the MOOC
policy established
 Level of su...
 Every institution should always have a roadmap to
improve and extend the use of current courses, such
as MOOCs:
1. facil...
christian.stracke@ou.nl
esther.tan@ou.nl
@ChrMStracke
@Taneste
www.opening-up.education
Let’s collaborate!
Thank you!
Your questions?
To be continued…
Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution ...
First References for GMQS
Stracke, C. M., et al. (2018). Gap between MOOC designers' and MOOC
learners' perspectives on in...
The Quality Reference Framework
Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation
Dimension 2:...
MOOQ project:
www.MOOC-quality.eu
Online community for the
Quality Reference Framework:
www.MOOC-quality.net
Session Title:
Minds-on session
on Design
Processes for
MOOCs
Facilitators:
Christian M. Stracke
Cleo Sgouropoulou
Nikos P...
Session 1:
Minds-on session on Design
Processes for MOOCs
Insights on:
• The Quality Reference
Framework in practice,
• De...
Upcoming SlideShare
Loading in …5
×

of

2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 1 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 2 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 3 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 4 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 5 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 6 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 7 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 8 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 9 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 10 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 11 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 12 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 13 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 14 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 15 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 16 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 17 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 18 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 19 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 20 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 21 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 22 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 23 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 24 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 25 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 26 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 27 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 28 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 29 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 30 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 31 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 32 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 33 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 34 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 35 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 36 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 37 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 38 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 39 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 40 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 41 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 42 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 43 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 44 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 45 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 46 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 47 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 48 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 49 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 50 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 51 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 52 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 53 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 54 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 55 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 56 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 57 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 58 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 59 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 60 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 61 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 62 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 63 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 64 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 65 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 66 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 67 2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan Slide 68
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan

Download to read offline

2018-07-13 Presentation at European MOOQ Conference in Athens on MOOQ and the Quality of MOOCs - Findings and Tools by Christian M. Stracke and Esther Tan from OUNL

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - Findings and Tools Stracke Tan

  1. 1. MOOQ and the Quality of MOOCs: Findings & Tools Christian M. Stracke & Esther Tan
  2. 2. Open University of the Netherlands Global cooperation: ECNU & KNOU Global initiative ICORE for OR & OE International WLS / LINQ Conference eLC European Institute ICDE Chair in OER Dr. Christian M. Stracke: Open Learning & Education, Innovations, Policies, Quality & Competences, Impact
  3. 3. Open University of the Netherlands The Quality of MOOCs Let’s Learn to Learn Seamless Learning Dr. Esther Tan Technology-Enhanced Learning, Innovations in & out Classroom
  4. 4. Presentation Outline 1. The Quality of MOOCs: What is the problem? 2. A Conceptual Framework towards MOOC QRF: Theoretical & Methodological Approach 3. Findings from MOOC surveys: Learners, Designers & Facilitators 4. Findings from MOOC OEQ: Learners, Designers & Facilitators 5. Findings from MOOC semi-structured interviews: Providers
  5. 5. Any discussion on the quality of MOOCs should consider the goals of both, the MOOC learner and the provider. (Hayes, 2015) Critical questions to be addressed:  Who are the MOOC users, and why?  What makes a good MOOC from the learners’ experiences with MOOCs?  What are the best design principles and best practices as indicators of quality?  Are these quality indicators also MOOC domain specific and/or MOOC type specific? Problem Statement
  6. 6. A Research Framework for MOOQ “When one designs any course, one has to have some learner cohort in mind.” (Macleod et al., 2015, p. 9) Main research goals: 1. Establish a research framework for the subsequent analysis of MOOC design patterns and best practices 2. Develop a Quality Reference Framework (QRF) for MOOCs Theoretical Framework
  7. 7. Global MOOC Quality Survey
  8. 8. Quality Reference Framework with criteria & checklist for MOOC design Our main goal is the collaboration with all to improve Open Education & MOOCs www.MOOC-quality.eu
  9. 9.  Subjects of Investigation: Learners, Designers, Facilitators & Providers  Mixed Methods Approach: Quantitative & qualitative data 1. Global MOOC Quality Survey, 2. Open-Ended questions (OEQ) & 3. Semi-structured interviews Methodological Framework
  10. 10. Instruments of Measure MOOC Learners MOOC Designers MOOC Facilitators MOOC Providers TOTAL Global MOOC Quality Survey 166 (69 qns.) 68 (89 qns.) 33 (58 qns.) - 267 Open- ended Questions 118 (4 qns.) 42 (4 qns.) 27 (4 qns.) - 185 Semi- structured interviews - 12 (15 qns.) 12 (10 qns.) 12 (13 qns.) 36
  11. 11. MOOC Survey Constructs Constructs Learners Designers Facilitators Experience with MOOC X X X Learning Objectives X X X Duration and Structure X X Learning Resources X X X Accessibility and Inclusion X X Learning Progress X Learning Environment X X Learning Assessment X X X Learning Certification X X Design Process X Pedagogical Decisions X Learning Support  Feedback & Facilitation  Interaction & Collaboration X X X
  12. 12. Global MOOC Quality Survey (GMQS)
  13. 13. Global MOOC Quality Survey
  14. 14. Demographic Profile Age range of all survey participants (learners, designers & facilitators) by gender
  15. 15. Demographic Profile Educational level of all survey participants (learners, designers & facilitators) by gender
  16. 16. n VB B N G VG Learning experience 166 4 4 13 75 70 Learning Experience (Learners)
  17. 17. n VB B N G VG Design experience 68 1 2 13 33 19 Design Experience (Designers)
  18. 18. Interaction from Learners‘ Perspective n N/A SD D N A SA LF 146 20 5 13 48 37 23 LL 146 15 3 17 34 51 26 LR 146 9 2 8 25 61 41 GG 146 37 4 15 50 24 16 Note: LF: Interaction between learners and facilitators LL: Interaction among learners LR: Interaction between learners and learning resources GG: Interaction among teams and groups
  19. 19. n R2 M2 p LF by learners 125 .094 9.382 .000*** LL by learners 130 .101 10.818 .000*** LR by learners 136 .112 12.286 .000*** GG by learners 108 .045 4.131 .026* Bivariate Correlations between LLR4 and LLE4
  20. 20. Interaction from Designers‘ Perspective n N/A SD D N A SA LF 52 2 1 5 11 24 9 LL 52 1 1 3 11 19 17 LR 52 3 1 0 4 22 22 GG 52 8 2 10 14 13 5 Note: LF: Interaction between learners and facilitators LL: Interaction among learners LR: Interaction between learners and learning resources GG: Interaction among teams and groups
  21. 21. n R2 M2 p LF by designers 49 .003 0.109 .703 LL by designers 50 .043 1.595 .143 LR by designers 48 .046 1.537 .138 GG by designers 43 .001 0.038 .821 Bivariate Correlations between DLR4 and DDE4
  22. 22. Interaction from Facilitators‘ Perspective n N/A SD D N A SA LF 32 0 1 3 3 15 10 LL 32 1 0 3 2 11 15 LR 32 0 0 0 5 12 15 GG 32 5 4 3 3 9 8 Note: LF: Interaction between learners and facilitators LL: Interaction among learners LR: Interaction between learners and learning resources GG: Interaction among teams and groups
  23. 23. n R2 M2 p LF by fa- cilitators 30 .181 0.242 .015* LL by fa- cilitators 29 .021 0.296 .435 LR by fa- cilitators 30 .030 0.287 .342 GG by fa- cilitators 25 .000 0.305 .971 Bivariate Correlations between FLR4 and FDE4
  24. 24. Open-ended Questions (Learners)
  25. 25. No of respondents across the six domains Domain Learners Designers Facilitators Social Sciences, Humanities & Law 24 9 2 Education & Lifelong Learning 19 12 18 Computing & Informatics 18 5 2 Science, Math & Engineering 16 5 1 Nature, Environment & Health 21 6 1 Business, Management & Economics 20 5 3 Total 118 42 27
  26. 26. MOOC Open-ended Questions (OEQ) MOOC Learners Q1. What were the three main strengths of the MOOC? Q2. What were the three main weaknesses of the MOOC? Q3. What was missing in the MOOC? Q4. Looking ahead into the development of this type of learning experiences, what could be improved in future MOOCs?
  27. 27. Age Range of MOOC Learners 18-24 25-34 35-44 45-54 55-64 65-74 >75 Total Business, Management & Economics 0 3 6 9 1 1 0 20 Nature, Environment & Health 1 5 4 1 6 4 0 21 Science, Math & Engineering 0 3 4 6 0 3 0 16 Computing & Informatics 1 4 5 4 4 0 0 18 Education & Lifelong Learning 1 1 7 3 5 2 0 19 Social Sciences, Humanities &Law 1 4 6 6 4 2 1 24 Total 4 20 32 29 20 12 1 118 Demographics of MOOC Learners
  28. 28. Educational Level of Learners No School- ing High school Bachelor’s degree Masters degree Doctorate degree Total Business, Management & Economics 0 1 4 10 5 20 Nature, Environment & Health 0 2 6 12 1 21 Science, Math & Engineering 0 1 4 7 4 16 Computing & Informatics 0 0 3 9 6 18 Education & Lifelong Learning 1 0 4 5 9 19 Social Sciences, Humanities &Law 0 0 4 12 8 24 Total 1 4 25 55 33 118 Demographics of MOOC Learners
  29. 29. Gender of MOOC Learners Male Female Other Total Business, Management & Economics 12 8 0 20 Nature, Environment & Health 14 7 0 21 Science, Math & Engineering 7 9 0 16 Computing & Informatics 8 10 0 18 Education & Lifelong Learning 11 8 0 19 Social Sciences, Humanities &Law 10 13 1 24 Total 62 55 1 118 Demographics of MOOC Learners
  30. 30. Learning Experience Very Bad Bad Neutral Good Very Good Total Business, Management & Economics 1 0 2 9 8 20 Nature, Environment & Health 0 0 0 9 12 21 Science, Math & Engineering 0 0 1 2 13 16 Computing & Informatics 1 1 1 8 7 18 Education & Lifelong Learning 1 1 0 9 8 19 Social Sciences, Humanities &Law 0 0 1 15 8 24 Total 3 2 5 52 56 118 Open-ended Questions (MOOC Learners)
  31. 31. Open-ended Questions (MOOC Learners) % of Activities Completed None <25% =50% >75% >100% Total Business, Management & Economics 1 2 2 3 12 20 Nature, Environment & Health 0 5 2 4 10 21 Science, Math & Engineering 0 1 3 3 9 16 Computing & Informatics 2 3 1 4 8 18 Education & Lifelong Learning 1 3 2 5 8 19 Social Sciences, Humanities &Law 1 2 3 9 9 24 Total 5 16 13 28 56 118
  32. 32. Open-ended Questions (MOOC Learners) Q1. What were the three main strengths of the MOOC? 0 10 20 30 40 50 60 70 80 90 Curriculum Design & Delivery Instructional Design & Technology Assessment & Evaluation Facilitation & Feedback Interaction & Collaboration NumberofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computing and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  33. 33. Open-ended Questions (MOOC Learners) Q2. What were the three main weaknesses of the MOOC? 0 5 10 15 20 25 30 35 Curriculum Design Instructional Design & Technology Assessment & Evaluation Faciliation & Feedback Interaction & Collaboration NumberofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computing and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  34. 34. Open-ended Questions (MOOC Learners) Three main strengths and weaknesses of the MOOC Strengths Weaknesses Curriculum Design & Delivery • Good choice & quality of content • Good teachers, presenters & tutors • LO aligns with content • Weak choice & quality of content • Short duration Instructional Design & Technology • Good integration of IT & media • Support self-regulation & individual learning paths • Poor use of IT technological tools • Resources lack variety & quality Interaction & Collaboration • Encourage local group discussion & activities • Foster interaction with field experts • Lack interaction: learner-tutor • No support for community building
  35. 35. Open-ended Questions (Designers)
  36. 36. MOOC Open-ended Questions (OEQ) MOOC Designers Q1. Which were the main design decisions that you made during the development of the MOOC that later proved to be successful? Q2. Which were the three biggest difficulties that you faced when designing the MOOC? Q3. Which design decisions did not pay off as you expected? Q4. Looking ahead into the development of this type of learning design experiences, what methods and tools could be helpful to improve the design of future MOOCs?
  37. 37. Age Range of MOOC Designers 25-34 35-44 45-54 55-64 65-74 >75 Total Business, Management & Economics 1 1 3 0 0 0 5 Nature, Environment & Health 1 3 0 1 0 1 6 Science, Math & Engineering 1 1 1 1 1 0 5 Computing & Informatics 0 1 4 0 0 0 5 Education & Lifelong Learning 1 2 5 3 1 0 12 Social Sciences, Humanities &Law 1 1 1 4 1 1 9 Total 5 9 14 9 3 2 42 Demographics of MOOC Designers
  38. 38. Educational Level of Designers High school Bachelor’s degree Masters degree Doctorate degree Total Business, Management & Economics 0 0 4 1 5 Nature, Environment & Health 0 0 3 3 6 Science, Math & Engineering 0 0 1 4 5 Computing & Informatics 1 0 1 3 5 Education & Lifelong Learning 0 0 3 9 12 Social Sciences, Humanities &Law 0 2 3 4 9 Total 1 2 15 24 42 Demographics of MOOC Designers
  39. 39. Gender of MOOC Designers Male Female Total Business, Management & Economics 3 2 5 Nature, Environment & Health 2 4 6 Science, Math & Engineering 2 3 5 Computing & Informatics 2 3 5 Education & Lifelong Learning 5 7 12 Social Sciences, Humanities &Law 6 3 9 Total 20 22 42 Demographics of MOOC Designers
  40. 40. Open-ended Questions (MOOC Designers) No. of MOOCs Designed 1 2 - 4 5 - 9 >10 Total Business, Management & Economics 3 2 0 0 5 Nature, Environment & Health 2 1 2 1 6 Science, Math & Engineering 1 2 2 0 5 Computing & Informatics 3 2 0 0 5 Education & Lifelong Learning 4 5 1 2 12 Social Sciences, Humanities &Law 3 4 2 0 9 Total 16 16 7 3 42
  41. 41. Design Experience Very Bad Bad Neutral Good Very Good Total Business, Management & Economics 0 0 0 5 0 5 Nature, Environment & Health 1 0 1 3 1 6 Science, Math & Engineering 0 0 1 2 2 5 Computing & Informatics 0 0 1 3 1 5 Education & Lifelong Learning 0 0 4 5 3 12 Social Sciences, Humanities &Law 0 1 0 3 5 9 Total 1 1 7 21 12 42 Open-ended Questions (MOOC Designers)
  42. 42. Open-ended Questions (MOOC Designers) Q1. Which were the main design decisions that you made during the development of the MOOC that later proved to be successful? 1 3 3 42 1 1 3 3 6 5 4 0 4 8 12 16 20 24 Assessibility& Inclusion Assessment & Evaluation Interaction & Collaboration Certification& Accreditation Curriculum Design& Delivery Expertise & Manpower Feedback & Facilitation Instructional Design& Technology No.ofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computer and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  43. 43. Open-ended Questions (MOOC Designers) Q2. Which were the three biggest difficulties that you faced when designing the MOOC? 3 3 1 3 3 2 8 1 2 2 1 1 1 2 3 5 4 4 2 5 3 0 4 8 12 16 20 24 Assessment & Evaluation Curriculum Design& Delivery Expertise & Manpower Feedback & Facilitation Institutional Support & Funding Instructional Design& Technology Interaction & Collaboration Open Access, Copyrights & Licensing No.ofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computer and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  44. 44. Open-ended Questions (MOOC Designers) Successful Decisions Biggest Challenges Curriculum Design & Delivery • Content delivery format • Content structure & LOs • Weak choice & quality of content • Short duration Instructional Design & Technology • Choice of learning activities • Integration of IT & media • Platform, software & production decisions Interaction & Collaboration Expertise & Manpower • Creating community of learners • Foster interaction between learner & tutor/ facilitator • Gap in content & instructional design knowledge • Coordination & collaboration, e.g., different experts & teaching staff Three main successful decisions and three biggest challenges
  45. 45. Open-ended Questions (Facilitators)
  46. 46. MOOC Open-ended Questions (OEQ) MOOC Facilitators Q1. Which were the main decisions that you made during the facilitation of the MOOC that later proved to be successful? Q2.Which were the three biggest difficulties that you faced when facilitating the MOOC? Q3 Which facilitation decisions did not pay off as you expected? Q4 Looking ahead into the development of this type of learning experiences, what methods and tools could be helpful to improve the facilitation of future MOOCs?
  47. 47. Age Range of MOOC Facilitators 25-34 35-44 45-54 55-64 65-74 >75 Total Business, Management & Economics 0 1 0 0 0 2 3 Nature, Environment & Health 0 0 0 1 0 0 1 Science, Math & Engineering 0 1 0 0 0 0 1 Computing & Informatics 1 0 0 1 0 0 2 Education & Lifelong Learning 0 6 6 3 3 0 18 Social Sciences, Humanities &Law 0 0 1 1 0 0 2 Total 1 8 7 6 3 2 27 Demographics of MOOC Facilitators
  48. 48. Educational Level of MOOC Facilitators High school Bachelor’s degree Masters degree Doctorate degree Total Business, Management & Economics 0 0 1 2 3 Nature, Environment & Health 0 0 0 1 1 Science, Math & Engineering 0 0 0 1 1 Computing & Informatics 1 0 1 0 2 Education & Lifelong Learning 0 4 8 6 18 Social Sciences, Humanities &Law 0 0 2 0 2 Total 1 4 12 10 27 Demographics of MOOC Facilitators
  49. 49. Gender of MOOC Facilitators Male Female Total Business, Management & Economics 0 3 3 Nature, Environment & Health 1 0 1 Science, Math & Engineering 1 0 1 Computing & Informatics 0 2 2 Education & Lifelong Learning 10 8 18 Social Sciences, Humanities &Law 0 2 2 Total 12 15 27 Demographics of MOOC Facilitators
  50. 50. Open-ended Questions (MOOC Facilitators) No. of MOOCs Facilitated (OEQ) 1 2 - 4 5 - 9 >10 Total Business, Management & Economics 2 0 0 1 3 Nature, Environment & Health 0 0 0 1 1 Science, Math & Engineering 1 0 0 0 1 Computing & Informatics 1 1 0 0 2 Education & Lifelong Learning 6 7 2 3 18 Social Sciences, Humanities &Law 0 2 0 0 2 Total 10 10 2 5 27
  51. 51. Facilitation Experience Good Very Good Total Business, Management & Economics 3 0 3 Nature, Environment & Health 0 1 1 Science, Math & Engineering 1 0 1 Computing & Informatics 1 1 2 Education & Lifelong Learning 12 6 18 Social Sciences, Humanities &Law 1 1 2 Total 18 9 27 Open-ended Questions (MOOC Facilitators)
  52. 52. 0 1 2 3 4 5 6 7 8 9 Provide guidelines on attendance, quiz & test Monitor peer review process & forum discussion Show presence & provide feedback Leverage technological tools to support e-learning Promote interaction & group dynamic Provide quality content & appropriate pedagogical approach No.ofComments Successful Facilitation Decisions Social Sciences, Humanities & Law Education & Lifelong Learning Computing & Informatics Science, Math & Engineering Nature, Environment & Health Business, Management & Economics
  53. 53. 0 1 2 3 4 5 Difficult to engage & retain learners Integrate IT & Media to support learning Lack IT competencies (learners) Mismatch of content and target learners Overload learners with quizzes & assignments Provide free certfication and course Unable to foster interaction & collaboration No.ofComments Unsucessful Facilitation Decisions Social Sciences, Humanities & Law Education & Lifelong Learning Computing & Informatics Science, Math & Engineering Nature, Environment & Health Business, Management & Economics
  54. 54. Semi-Structured Interviews Questions (Providers)
  55. 55. Categories Descriptor Indicators Role of MOOCs in HE Providers perspective on the role of MOOCs in the current HE national and international scenario  Pedagogical innovation  Institutional culture: Toward hybrid & blended courses; teaching internally & reaching out externally Reasons to provide MOOCs Identification of causes for offering MOOCs • Institutional mission: to “place people” as the largest course provider in FutureLearn; foster openlearn in the eco system; educational research on MOOCs; public engagement to showcase Uni; capacity development within the Uni • Institutional goal: Foster innovation in digital learning; stay relevant in the changing educational ecosystems Summative Report of Coded Interviews
  56. 56. Categories Descriptor Indicators Partnerships Establishment, organization and management of partnerships  Partnership structure: Shared network with a core group of universities to share research project  Great collaboration between researchers from various disciplines (excellent model); good to have external driver to promote internal change; helped in best practices. Institutional Implementation Policies Theoretical design of the MOOC business model to implement  More strategic approach developed: lifelong learning, apprenticeship; MOOCs as experimentation spaces - separate from other learning & teaching platforms  Business model for delivering MOOCs: undergraduate and masters degrees) & chargeable certification (for some types) Summative Report of Coded Interviews
  57. 57. Categories Descriptor Indicators Institutional organizational strategies Description of the MOOC organization and development process in the institution  Support structures: 1. Set up a content team – work with partners to design well-designed courses with quality; 2. Uni put in place a number pedagogical coordinators to help teaching staff to prepare the course and provide ideas and experiences, technical staff for recording and post production, staff for subtitles. 3. Central unit works with individual academics to find the best plan/ fit  Re-train/ Assimilate teachers into the MOOC culture: provide teachers a checklist on dos & donts for content in an online learning environment. Summative Report of Coded Interviews
  58. 58. Categories Descriptor Indicators Sustainability Characterization of the return on investment  Funding source(s): 1. Business model: a number of bis model. Some funded by external org. most funded internally thru OU open learning budget (by media budget fr media unit) 2. Budget from uni for central unit plus some money from Edx (but not sufficient) received from Edx every 3 months some money; the objective is to improve education; not profit- making. 3. Four mio raised funding externally for the fully online masters study Summative Report of Coded Interviews
  59. 59. Categories Descriptor Indicators Evaluation Design of the evaluation process for the MOOC policy established  Level of success for the institution: 1. Grant awareness: revenue gains, paid courses 2. Prestige & research gains. 3. Educational success: moving from standard to hybrid & blended learning in accredited courses 4. Community of learners: with other likeminded people/ other Uni 5. Met objectives, research, Uni promotion & capacity building  Specialist evaluation apart from usual questionnaire on institutional e-learning and pedagogical innovation pedagogical model Summative Report of Coded Interviews
  60. 60.  Every institution should always have a roadmap to improve and extend the use of current courses, such as MOOCs: 1. facilitates implicit social learning, and may also help to attract new students to programmes. 2, the sociocultural nature of students is changing in Europe and beyond so it is important for to undertake multicultural adaptation of courses for the different student types 3. ECTS for MOOCs  Globalization strategies of the training spaces: MOOCs raise new research questions/problems that deserve further study (e.g., what are the consequences of these new models to knowledge sharing/building on a global level). Summative Report of Coded Interviews Final words from MOOC Provider
  61. 61. christian.stracke@ou.nl esther.tan@ou.nl @ChrMStracke @Taneste www.opening-up.education Let’s collaborate!
  62. 62. Thank you! Your questions? To be continued…
  63. 63. Open CC License for sharing & re-using slides This work is free to share under the creative commons licence: "Attribution – Noncommercial – Share Alike 3.0" You can copy, distribute and transmit the work under the following conditions: 1. Attribution – 2. Noncommercial – 3. Share Alike Licence: Attribution-Noncommercial-Share Alike Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/
  64. 64. First References for GMQS Stracke, C. M., et al. (2018). Gap between MOOC designers' and MOOC learners' perspectives on interaction and experiences in MOOCs: Findings from the Global MOOC Quality Survey. In M. Chang, N.-S. Chen, R. Huang, Kinshuk, K. Moudgalya, S. Murthy, & D. G. Sampson (Eds.), Proceedings 18th IEEE International Conference on Advanced Learning Technologies (ICALT) (pp. 1-5). IEEE: Computer Society. DOI 10.1109/ICALT.2018.0000 Stracke, C. M., & Tan, E. (2018). The Quality of Open Online Learning and Education: Towards a Quality Reference Framework for MOOCs. In J. Kay, & R. Luckin (Eds.), Rethinking learning in the digital age. Making the Learning Sciences Count: The International Conference of the Learning Sciences (ICLS) 2018 (pp. 1029-1032). London: ISLS. Stracke, C. M. et al. (2017). The Quality of Open Online Education: Towards a Reference Framework for MOOCs. In Proceedings of 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 1712-1715). IEEE Xplore. DOI: 10.1109/EDUCON.2017.7943080 To be continued …
  65. 65. The Quality Reference Framework Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation Dimension 2: Perspectives Pedagogical, Technological, and Strategic Dimension 3: Roles Designer, Facilitator, and Provider
  66. 66. MOOQ project: www.MOOC-quality.eu Online community for the Quality Reference Framework: www.MOOC-quality.net
  67. 67. Session Title: Minds-on session on Design Processes for MOOCs Facilitators: Christian M. Stracke Cleo Sgouropoulou Nikos Palavitsinis Session Title: Dealing with practical quality problems when running a MOOC Facilitators: Bill Vassiliadis Antonia Stefani Esther Tan Achilles Kameas Parallel Sessions
  68. 68. Session 1: Minds-on session on Design Processes for MOOCs Insights on: • The Quality Reference Framework in practice, • Design decisions, • Roles of designers, facilitators and providers Room: 3rd floor Session 2: Dealing with practical quality problems when running a MOOC Insights on: • Drop-out rates, • Failure to collaborate, • Fairness in automatic assessment Room: here (1st floor) Parallel Sessions

2018-07-13 Presentation at European MOOQ Conference in Athens on MOOQ and the Quality of MOOCs - Findings and Tools by Christian M. Stracke and Esther Tan from OUNL

Views

Total views

228

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

2

Shares

0

Comments

0

Likes

0

×