The document discusses surveys conducted to evaluate learners' experiences in MOOCs offered by UNSW Australia. It finds that surveys embedded within course activities, like pre-course surveys, obtain higher response rates than stand-alone surveys. In-video surveys are more effective than those outside videos at capturing engagement. Across four MOOCs, common motivations were personal growth, topic interest, and job relevance. Course B learners primarily aimed to develop teaching strategies. Engagement varied across activities, with videos and forums most used. Paying learners reported higher perceptions of video usefulness than non-paying learners.

1. Evaluating the ‘Student’ Experience in
Massive Open Online Courses (MOOCs)
Catherine Zhao & Lorenzo Vigentini
Learning and Teaching Unit, UNSW Australia

2. In this presentation -
• Background – Previous work on Student learning
Experience & UNSW MOOCs
• Methods - the surveys for evaluation
• Findings -
o Surveys - fit-for-purpose?
o Learners – what’s it like to experience a MOOC?
• Current work & Challenges

3. A Model of the Student Learning Experience
Worth of
Experience
Future
Past
Motivation
Experience
Perception
Engagement
Aspirations
Expectations
Personal
development
‘match’ or ‘fit’ with
context/LE
Direct future
benefits
Vigentini, L., and Zhao, C. Revisiting end-of-semester evaluation: the questions, the instrument and the
process.,” in AAIR : turning silver into gold, Melbourne, 2014.

4. Background – Why UNSW joined the ‘bandwagon’

5. Background- What it takes to design and deliver a MOOC

6. Five known facts about MOOCs
Massive enrollments
Dramatic dropouts
Unconventional learners
(Relatively) limited design/delivery options
Engagement and the learner experience
The rise of ‘certificates’

8. Background - Four MOOCs we look at
A B C D
Disciplines Engineering Education Science Medicine
Duration (weeks) 9 6 9 8
Target Group Engineers Teachers
High school
students/ teachers
General public
Course Design All-at-once All-at-once Sequential Semi-sequential
Course Delivery All-at-once Staggered Staggered Staggered
Videos 110 224 98 58
Quizzes 10 22 42 2
Assignments 7 3 2 2
Forums 54 105 63 64
Registrants (Active
learners*)
32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%)
*Active learner: those who appear at least once during the course
MOOCs – A = engineering, B = education, C = physics, D = medicine

9. Background- What & When Surveys
Evaluation Surveys Deployment A B C D
Demographic survey
Prior to course
commencement
   
Pre-course survey OR
First course activity
   
In-video surveys (IVS)
During
the
course

Quick evaluation (Video) 
Quick evaluation (Module) 
Quick Question survey  
Post-course survey End of course    
 Present in MOOC
MOOCs – A = engineering, B = education, C = physics, D = medicine

10. Methods - Overview of the Pre-Course survey questions
Common A B C D Course-specific A B C D
Reasons/Purpose of Joining    Background-Gender, Ethnicity 
Prior Professional
Experience/Knowledge
    English Proficiency 
Intended Effort    Prior MOOC Experience 
Familiarisation with Topics
 
Geographic Location 
Academic Qualification   
Contributing Elements of
‘Teaching Online’

Industry of Employment

 Defining ‘Completion' 
Preventing Factors for Course
Completion
 
Learning Preference 
Expected Outcomes
(course related)
  Achievement Goal Oriented 
 Present in the course
MOOCs – A = engineering, B = education, C = physics, D = medicine

11. Common A B C D Course-specific A B C D
Course Satisfaction 
   Proportion of Completed Course
Activities

Course Features     Evaluating Peer Assessment 
Actual Effort 

  (Probing) Self-Regulated Learning 
Desired Topics   
Satisfaction on Personal Development
 

(Intend to) Apply Course Topics 
  
Considering Credit-Bearing Courses
 
Methods- Overview of the Post-course survey questions
MOOCs – A = engineering, B = education, C = physics, D = medicine

12. Methods– Pre- & Post-Course Survey Sample Sizes
A B C D
# Active Learners 19708 18024 10576 4673
Required sample size* 2141 2119 1957 1587
Pre- & Post- Course survey
Pre-course survey sample 2684 4490 264 401
response rate 13.62% 24.91% 2.50% 8.58%
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
response rate 3.24% 1.74% 1.19% 1.52%
Margin of error 3.82% 5.49% 8.68% 11.54%
Coursera Demographic Survey
# of registrants 32928 28864 22487 13185
Demographic Survey Responses 2774 2873 2720 1466
Margin of error 1.20% 2.20% 1.52% 2.41%
*Required sample size is calculated under the condition of CL = 95%, and margin of error = 2%

13. Methods – Patterns of participation in Pre-& Post-
Course surveys
Counts of responses in the four MOOCs’ pre-course surveys (logarithmic scale) over 9 weeks.
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses A
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
B
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses
C
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
D
Pre Post

14. Methods- Collection tools
Course A IVS (In-video survey) Course C Quick Evaluation
MOOCs – A = engineering, B = education, C = physics, D = medicine

15. Research Questions
1) Are the tools used robust enough to characterize the
learners’ experience in MOOC?
2) Are there substantial differences in the modes of
gathering feedback?
3) Is it possible to identify key elements of the learners’
MOOC experiences?
4) What is the relevance of ‘teaching’ questions for the
online experience?

16. Results – Are the surveys robust?
• Accuracy
• A B C D
Required sample size (CI=95%,margin of error 2%) 2141 2119 1957 1587
Pre-course survey sample 2684 4490 264 401
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
Margin of error 3.82% 5.49% 8.68% 11.54%

17. Results – Are the surveys robust? (Cont’d)
A B C D
Motivation Block 0.78 0.86 0.65
Course experience Block 0.90 0.94 0.89 0.92
IVS items 0.82
Confidence 0.84
Satisfaction of personal gain 0.91
Satisfaction of course materials and
personal gain
0.92
(probing) self-regulated learning skills 0.90
• Reliability – Cronbach’s alpha
MOOCs – A = engineering, B = education, C = physics, D = medicine

18. Results – Are the surveys robust? (Cont’d)
• For the motivation block that is <.7
(D) Reason items
Cronbach's Alpha if
Item Deleted
I am taking the course out of general interest, curiosity, or
enjoyment.
.722
I am interested in taking a course from this particular
institution.
.616
I am interested in learning from the expert researchers
and clinicians delivering the course.
.636
I want to connect with other students interested in this
topic.
.589
The course supports my current academic program. .565
The course is aligned with my current job responsibilities
or company's line-of-business.
.588
The skills gained from this course may be useful for
obtaining a new job.
.571
MOOCs – A = engineering, B = education, C = physics, D = medicine

19. Results – Comparison of Evaluation Approaches
• The reasons for adopting different approaches
1. Technical constraints
2. Accommodating various requirements put
forward by academic leads
3. Test different tools/ways to evaluate
• Two key sets of approaches we are comparing -
1. First course activity in A & B vs. Pre-course
survey in C & D
2. In-Video Surveys (IVS) vs. Quick Evaluation (QE)
MOOCs – A = engineering, B = education, C = physics, D = medicine

20. Results – Effectiveness of Pre-course survey
• ‘Embedded’ (= first course activity) is more effective
than ‘stand-alone’
• Higher response rate
• measures how much learners want to engage
A B C D
13.62% 24.91% 2.50% 8.58%
MOOCs – A = engineering, B = education, C = physics, D = medicine

21. 1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
A - IVS
INTSE Views of
lecture videos
INTSE # Participation
in IVQs
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
C - Quick Evaluation P2P Views of lecture
videos
P2P # Participation in
Quick Evaluation
Results – In-Video-Surveys vs. Quick Evaluation
Inside the video is more effective than outside the video.
MOOCs – A = engineering, B = education, C = physics, D = medicine

22. Our learners are…
A B C D
Ave. Age (SD)
34.37
(11.37)
39.91
(12.39)
35.9
(15)
37.35
(13.84)
Gender ratio
(M/F)
4/1 1/1.2 2.6/1 1/1.2
Full-time
Employed
57% 63% 59% 59%
Full-time
Studying
41% 30% 39% 38%
Bachelors and
Above
74% 82% 63% 64%
Prior Experience 41.51% 57.20% - 79.25%
MOOCs – A = engineering, B = education, C = physics, D = medicine

23. Results – motivations I
A C D
M(SD) M(SD) M(SD)
For personal growth and enrichment 3.23 (.76) 2.10(.72)
General interest in topic 3.14 (.74) 2.23(.72) 3.23(.92)
Relevant to job 2.63 (1.01) 1.6(.76) 2.71(1.38)
For fun and challenge 2.58 (.96) 1.42(.63)
Earn a certificate/statement of
accomplishment
2.42 (1.03) 1.55 (.70)
Course offered by prestigious
university/professor
2.17 (1.03) 1.91(.77) 3.06(.97)
For career change 2.07 (1.06) 1.32(.59) 1.92(1.41)
Experience an online course 1.81 (.93) 1.21(.51)
Relevant to school or degree program 1.75 (1.00) 1.74(.83) 2.56(1.37)
To improve my English skills 1.66 (.95) 1.52(.71)
Relevant to academic research 1.65 (.98) 2.37(.86)
Meet new people 1.52 (.76) 1.65(.70) 1.96(1.06)
Take with colleagues/friends 1.32 (.68) 1.93(.75)
1=Not at all important, 2 = somewhat important, 3 = very important, 4 = extremely important

24. Results – Motivations II
Reasons
Item loading
A1
C2
D3
I II III VI I II III VI I II
Experience an online course1,2 0.7 0.7
Meet new people1,2,3 0.7 0.8 0.6
To improve my English skills1,2 0.7 0.2 0.6
Take with colleagues/friends1,2 0.7 0.6
Course offered by prestigious
University/Professor1,2,3
0.6 0.8 0.8
For fun and challenge1,2 0.8 0.7
Personal growth&enrichment1,2 0.8 0.8
General interest in topic1,2,3 0.7 0.8 0.5
Relevant to academic research1,2 0.9 0.8
Relevant to academic study1,2,3 0.9 0.8 0.8
Relevant to job1,2,3 0.8 0.7 0.8
Earn a certificate1,2 0.6
For career change1,2,3 0.5 0.8 0.7
Items with loading lower than 0.5 are removed. Superscripts 1, 2, 3 represent occurrence in A, C, & D respective.
MOOCs – A = engineering, B = education, C = physics, D = medicine

25. Results – Course B Specifics
Intent & Goals

26. Results – Course B Specifics (Cont’d)
2.20%
2.77%
3.72%
3.77%
8.76%
14.17%
30.43%
34.17%
I want to develop my knowledge and
understanding of a specific concept
I want to exchange ideas and learn from
colleagues
I am interested in looking at the course design
I am curious about the topic
I want to complete the course and obtain
recognition and certification for my work
I want to use my knowledge to develop a personal
online teaching strategy
I want to develop my knowledge and
understanding of the overall topic
I want to develop an online learning design that I
can use in my own teaching
(N = 4083)
• Overview of Learners’ Intents

27. Results – Course B Specifics (Cont’d)
• Intent is not sig. correlated to learners’ behaviours
• Data Mining (K-NN, Naïve Bayes) using learners’ characteristics (incl.
behaviours) indicate 30% predictive accuracy for performance class (no
grade, pass, distinction).

28. Results – Course B Specifics (cont’d)
Activity by intent

29. Results – Course B Specifics (Cont’d)
Week 2
Week 4
Week 7
Assessment scores vs. Intents

30. 0
10000
20000
30000
40000
50000
60000
Announcement
Times
Site Access
Video Views &
Downloads
Announcements superimposed on User Video and Site Access Activity
Results – Course B Specifics (Cont’d)

31. Results – The same in Course D

32. Results – Engagement with Forum
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine

34. Results – Engagement with Quizzes
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine

35. Results – Engagement with Videos
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine

36. Results – Engagement III
Comparing paying & non-
paying learners
F t df
Sig.
2-tails
Mean
Diff’
CI=95% Interval
of diff'
Lower Upper
Q1 I find this lecture useful.
33.66 -4.9 9092 <0.05 -0.11 -0.16 -0.07
Q2 I understand the content
of this lecture. 24.81 -3.89 9045 <0.05 -0.09 -0.13 -0.04
Q3 I’d like to explore other
modules of this course.
43.62 -5.16 8896 <0.05 -0.11 -0.15 -0.07
Results of F-test suggest unequal variance hence the t-test results assumed unequal variance are used.
(Using Course A as an example – in-video survey items)
MOOCs – A = engineering, B = education, C = physics, D = medicine

37. Results – Learners’ Course Experience
Course experience items (mapped on new CATEI)
A B C
I II I II I II
Overall, I am satisfied with the quality of this course. 0.9 0.7 0.6
The material of the course was presented in an engaging manner. 0.9 0.8 0.8
Overall, this course met my expectations. 0.9 0.8 0.6
Examples, illustrations or real-world cases were used effectively to
explain things.
0.9 0.8 0.7
The course encouraged my interest in this field of study. 0.8 0.8
The goals and requirements of the course were made clear to me. 0.8 0.8 0.7
Overall I have improved the knowledge/skills I will need. 0.8 0.8 0.8
Quizzes helped me to evaluate my progress effectively. 0.6 0.6 0.8
Interacting in the forums helped me to clarify things I did not
understand.
0.9 0.9 0.6
Items with loading lower than 0.5 are removed.
MOOCs – A = engineering, B = education, C = physics, D = medicine

38. Course experience Items
D
I II
Depth of content 0.8
Course materials 0.8
Lecture videos 0.8
Clarity of explanations 0.8
Course content 0.8
Knowledgeability 0.8
Instructor/teaching staff 0.8
Overall satisfaction 0.7
In-video quizzes 0.7
Presentation skills 0.5
Assessments 0.9
Discussion forums 0.6
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.
a. Rotation converged in 3 iterations.
Results – Learners’ Course Experience (Cont’d)
MOOCs – A = engineering, B = education, C = physics, D = medicine

39. Results – the Worth of Course Experience
R2 F Sig.
A 0.17 1.22 0.26
C 0.79 0.72 0.72
D 0.39 1.08 0.38
• Motivation cannot predict satisfaction
LinearRegression(motivations) -> ‘How satisfied am I’
MOOCs – A = engineering, B = education, C = physics, D = medicine

40. Aspects
(Question Block) A coefficient Sig'
Course
Experience
Overall, this course met my expectations 0.54 0.00
The material of the course was presented in an engaging manner 0.19 0.00
The course encouraged my interest in its field of study 0.11 0.00
Examples, illustrations or real-world cases were used effectively to explain
things
0.08 0.01
The goals and requirements of the course were made clear to me 0.07 0.02
Aspects
(Question Block B coefficient Sig'
Course
Experience
Overall this course met my expectations 0.57 0.00
Satisfaction Provided an opportunity for fun and challenge 0.10 0.03
What can predict satisfaction
‘How satisfied am I’ <= LinearRegression(multiple items)
Results – the Worth of Course Experience (Cont’d)

41. Results – the Worth of Course Experience (Cont’d)
Aspects
(Question Block)
C coefficient Sig'
Overall I have improved the knowledge/skills I will need 0.63 0.00
Course Experience The course encouraged my interest in this field of study 0.61 0.01
The material of the course was presented in an engaging manner 0.32 0.02
The course encouraged my interest in this field of study 0.36 0.02
The experimental tasks were useful to apply the theory 0.30 0.02
LinearRegression(multiple items) ->‘How satisfied am I’
Aspects
(Question Block)
D coefficient Sig'
Self-regulation
I am confident that I could deal efficiently with unexpected events. 0.87 0.00
I can remain calm when facing difficulties because I can rely on my
coping abilities. 0.62 0.01
I can solve most problems if I invest the necessary effort. 0.35 0.02
I can usually handle whatever comes my way 0.32 0.05
Course Experience
Discussion forums 0.42 0.01
In-video quizzes 0.45 0.03
Depth of content 0.67 0.03
I found the material interesting. 0.31 0.03
Motivation
I am taking the course out of general interest, curiosity, or
enjoyment.
0.28 0.03

42. Conclusions
1. HBR paper*: two types of learners – ‘career
builders’ and ‘education seekers’.
2. Motivation weakly related to engagement or
satisfaction
3. Some relations between behaviors (engagement)
and performance
4. ‘Embedded’ evaluation tools are more effective
5. Mapping of ‘online’ learning experience
* Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting
from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-benefiting-from-
moocs-and-why

43. Challenges
What are the characteristics of MOOCs that scaffolds
learning?
The intricacy of the combination of the factors which
contribute to the worth of course experience.
The matter of Agency.

44. Questions and feedbacks
catherine.zhao@unsw.edu.au