SlideShare a Scribd company logo
Learning Analytics & Data Science Research Group – 21 June 2016
Different Sides of MOOCs – Comparing
Perceptions and Experiences across Platforms
Catherine Zhao & Lorenzo Vigentini
Outline
• Recap of the ‘flipped’ paper
• Key features of Coursera & FutureLearn (FL)
• In Coursera & FL
1. Various feedback gathering means
2. MOOCs Learners’ demographic profiles
3. Elements influencing MOOCs Learners’ experiences
4. The overall satisfaction with experience
Recap of ‘Evaluating the student experience in
MOOCs’
http://bit.ly/flipMoocExp
The context of the paper - four Coursera MOOCs
http://bit.ly/flipMoocExp
INTSE LTTO P2P PMED
Disciplines Engineering Education Science Medicine
Duration (weeks) 9 6 9 8
Target Group Engineers Teachers
High school
students/ teachers
General public
Course Design All-at-once All-at-once Sequential Semi-sequential
Course Delivery All-at-once Staggered Staggered Staggered
Videos 110 224 98 58
Quizzes 10 22 42 2
Assignments 7 3 2 2
Forums 54 105 63 64
Registrants (Active
learners*)
32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%)
*Active learner: those who appear at least once during the course
Highlight 1- Design drives learning to some extent Example–
engagement with quizzes
INTSE
P2P
PersMed
LTTO
Highlight 2. in-context works better to collect data
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
INTSE Views of
lecture videos
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
P2P Views of
lecture videos
Highlight 3-Model of evaluating mainstream SE could also be
applied in MOOCs
Course Enablers are the key
features of the educational
platform, the course designs
and mode of teaching to
support learning.
Precursors are students’ prior
academic and professional
experience, motivations,
aspirations and expectations
of learning.
Engagement is the totality
of learners’ interactions with
the platform, peers, and the
course.
The Worth of Experience is what
learners take away and project to their
future, evaluating the learning
experience, their satisfaction with the
course, the perceived utility and
effectiveness of the course for their
learning, and/or for the development of
knowledge and skills.
Conceptual building blocks characterizing the student experience
– Instructivist approach (Vigentini &
Clayphan, 2015)
– Mastery learning (Coursera
principles*)
– Networked/connected
learning (FL principles**)
– Openness (FL principles)
Key principles of Coursera & FL platforms
Vigentini, L & Clayphan, A. (2015) Pacing through MOOCs: course design or teaching effect?
International conference of educational data mining, Madrid, Spain
*Coursera principles https://www.coursera.org/about/ **FL principles
https://www.futurelearn.com/about/our-principles
Platform content assessment activity Discussion
forum
Coursera Y Y Y Y
FL Y Kinda no Y Errrr…
Key feature of MOOCs on Coursera & FL
Coursera’s standard survey
instruments
+
‘In-house’ surveys by
offering institutions
Throughout offering period
FL’s standard pre- & post-
course surveys
Stand-alone
Throughout offering period
Feedback gathering in Coursera & FL
Feedback gathering in Coursera & FL (cont`d)
*Coursera standard survey
**Coursera no longer deployed
***Coursera no long supported for evaluation purposes
Deployment period
Evaluative Instruments in
Coursera
Evaluative Instruments in
FL
Prior to course commencement
Demographic survey* Pre-course Survey
Pre-course survey
End of course Post-course survey Post-course Survey
During the course
Quick evaluation (Video)
Quick evaluation (Module)
Quick Question survey**
In-video surveys (IVS)***
The pre- & post-course surveys in Coursera
Areas
Pre-course survey question key
words Area
Post-course survey question
key words
Motivation Reasons of joining Satisfaction Course Experience
expectation expected outcomes
Satisfaction of Personal gains
intended weekly hours Engagement Actual Effort
Perception of ‘Completion' Proportion of Completed
Course Activities
Preventing Factors for Course
Completion
Future aspiration (Intend to) Apply Course
Topics
Learning
preference
Consider the way you prefer to learn
Considering Credit-Bearing
Courses
Demographics Prior professional experience Evaluation of
course design
Peer Assessment
Familiarisation with Topics (specific) Course Features
Qualification
Self-efficacy Confidence and strategies of
problem solving
Background-Gender, Ethnicity
English Proficiency
Prior academic experience
Geographic Location
Personal goals
What goals have you set for yourself
in this course
Mapping of items between pre-course surveys in
FLPre-course survey version I Pre-course survey version II
hope to get out of this course? I am taking this course because…
Additional thoughts about your expectations of FutureLearn. Additional thoughts about your expectations of
FutureLearn.
Have you taken a course delivered mostly or fully online
before, including Massive Open Online Courses (MOOCs)?
Have you taken a course delivered mostly or fully online
before, including Massive Open Online Courses (MOOCs)?
What sort of online course have you taken? What sort of online learning have you participated in?
How did you find out about this course? How did you find out about this course?
Country Which of the following related roles have you held in the
past year?Age group
Gender
Employment status
Industry of employment?
Qualification
Do you have a disability, long-term health condition, mental
health condition, specific learning difficulty (such as dyslexia),
or other physical or mental impairment?
What previous experience, if any, do you have in this
subject area?
How you would like to learn on FL
Where do you expect to do the course
Subject areas of interested
The pre-course surveys in FL
Areas Pre-course survey version I Areas Pre-course survey version II
Expectations hope to get out of this course? I am taking this course because…
Additional thoughts about your
expectations of FutureLearn.
Additional thoughts about your
expectations of FutureLearn.
Prior exposure to online
learning
Have you taken a course delivered
mostly or fully online before, including
Massive Open Online Courses
(MOOCs)?
Have you taken a course delivered
mostly or fully online before,
including Massive Open Online
Courses (MOOCs)?
What sort of online course have you
taken?
What sort of online learning have
you participated in?
Marketing and
communication
How did you find out about this
course?
How did you find out about this
course?
Demographics Country Which of the following related roles
have you held in the past year?Age group
Gender
Employment status
Industry of employment?
Qualification
Health conditions Do you have a disability, long-term
health condition, mental health
condition, specific learning difficulty
(such as dyslexia), or other physical or
mental impairment?
Prior
experience
What previous experience, if any, do
you have in this subject area?
Preferences for learning How you would like to learn on FL
Where do you expect to do the course
subject of interest Subject areas of interested
The post-course surveys in FL
Areas Post-course survey question key words
Learning preferences
How you felt about learning on FutureLearn.
Where did you do the course?
Aspects of technology How easy was it to find your way around the site?
Expectation How did the course difficulty meet with your expectations?
To what extent did FutureLearn meet your expectations
Engagement Roughly how often did you visit the course?
How did you feel about the length of the course?
Evaluation of course design How you felt about the course design and content.
To what extent did you find the educator(s) engaging?
Your favourite part of the course
Your least favourite part of the course
How could the course be improved?
Value of a statement
How likely are you to purchase a Statement of Participation for this course?
Why are you interested in a Statement of Participation?
Please let us know why you are unlikely to purchase a Statement of Participation
What would improve a Statement of Participation?
(Satisfaction) of overall experience How would you rate your overall experience of the course?
Future aspiration
How will you pursue your interest in the subject now that the course is complete?
What would you like to study next?
Marketing and communication How likely would you be to recommend FL to a friend?
Related study focusing on SDL
Areas Key categories Concept
Personal learning identity
Time/planning
Leisure/casual learning
Health as a driver
1. Daily life elements affecting the
learning experience
Technology as driver (good/bad)
Learning mobility
2. Technical aspect affecting the
learning experience
Balance known & new
Changing learning pace
3. Actions undertaken to adapt
learning
Various items deployed across
questions
4. Perception of individual &
collaborative learning
An experience of self-directed
learning (Knowles 1975,
Merriam 2001, de Waard et al.
2015)
de Waard, Inge; Kukulska-Hulme, Agnes and Sharples, Mike (2015). Self-Directed Learning in Trial
FutureLearn courses. In: Proceedings Papers, EMOOCS, pp. 234–243.
FL Week/Module evaluation (for Adaptive tutorials)
Randomised/balanced item presentation (to reduce number
of questions) in 5 different areas:
• Reasons to enroll (ranking of 3 items)
• Engagement preferences (2 items)
• Satisfaction (2 overall + 2 items)
• Rating of adaptive learning (4 items)
• Successful completion preference (4 items)
Pre-course survey samples
Post-course survey samples
Coursera MOOC
# active
learners
Post-course survey
sample
response rate
Margin of
error
INTSE 19708 638 3.24% 3.82%
LTTO 18024 313 1.74% 5.49%
P2P 10576 126 1.19% 8.68%
HL 4673 71 1.52% 11.54%
FL MOOC
Total
registrants
# post-course survey
responses
Post-course
survey response
rate
Margin of
Error
WWI lessons legacy 9747 396 4.06% 4.82%
Military ethics 6218 102 1.64% 9.62%
Remaking nature 7907 127 1.61% 9.63%
Maths linear quadratic 8757 49 0.56% 13.96%
What’s the difference in feedback gathering? (Cont`d)
*Coursera standard survey
**Coursera no longer deployed
***Coursera no long supported for evaluation purposes
Deployment period
Evaluative Instruments in
Coursera
Evaluative Instruments in
FL
Prior to course commencement
Demographic survey* Pre-course Survey
Pre-course survey
End of course Post-course survey Post-course Survey
During the course
Quick evaluation (Video)
Quick evaluation (Module) Adaptive tutorial evaluation
Quick Question survey**
In-video surveys (IVS)***
The diversity of learners across 4 FL MOOCs
Demographic INTSE LTTO P2P PMED
Typical Learners
profile in a Cousera
MOOC*
Ave. Age (SD) 34.37 (11.37) 39.9 (12.41) 35.9 (15) 37.35 (14) 35
Gender ratio (M-F) 4.0-1.0 1-1.2 2.6-1 1-1.2 1.7-1.0
Full-time Employed 57% 63% 59% 59% -
Full-time Studying 41% 30% 39% 38% 27%
Bachelors and Above 74% 82% 63% 64% 70%
Prior exposure to
online learning
- - - - -
Military ethics Math_algebra World War I
Remaking
nature
Typical Learners
profile in a FL MOOC
Ave. Age (SD) 36-45 26-35 56-65 36-45
Gender ratio (M-F) 1.5-1.0 1.0-1.0 1.2-1.0 1-2.23
Full-time Employed 7% 4% 18% 5%
Full-time Studying 3% 2% 3% 2%
Bachelors and Above 14% 11% 40% 13%
Prior exposure to
online learning
67.85% 61.83% 75.64% 48.28%
Preferences for learning
- Preferred ways to engage with the course,
content and others
- Where/how accessing the course
- Learning strategies?
Distribution of responses for preferred ways to learn
(4 MOOCs, N= 5071)
% as per column, indicates the percentage distribution of a particular rating point i.e. of all
ratings over ‘strongly like’ 25% on ‘preferred to learn by reading text’.
Distribution of responses for preferred ways to learn
(split by previous online experience)
% as per column, split by previous online experience, indicates the percentage distribution of a
particular rating point i.e. of all ratings over ‘strongly like’ 17.12% with prior exposure to online
learning on ‘preferred to learn by reading text’.
Where did learners prefer to learn
Math_algebra military ethics
remaking
nature
World war I
lessons legacy
At work 9.93% 13.25% 11.68% 8.47%
At school / college / university 6.68% 6.53% 5.47% 2.64%
During my commute 2.04% 4.09% 2.86% 2.46%
At home at a desk 47.12% 39.55% 41.58% 48.72%
At home on the sofa 25.70% 25.51% 23.53% 29.45%
In a public place with internet
access 5.84% 7.32% 9.89% 5.03%
Out and about 2.69% 3.76% 4.98% 3.24%
N=5071
% as per column, indicates the percentage distribution of a particular rating point i.e. of all
math_algebra responses, 47.12% chose ‘at home at a desk’.
Motivations and Expectations
• What is expected from the course
• Reasons for taking the course
Pre Course – Expectation distribution (4 MOOCs)
% as per column, indicates the percentage distribution per MOOC. Note: people could choose
as many options as they like.
Pre Course – Motivation distribution (IFL/TEE)
% as per row, indicates the percentage distribution over the Likert Scale in each statement.
What are the emerging factors in motivation?
• Are the wording of the questions effective?
• Which questions are more closely correlated with each
other?
• Can we reduce the dimensions for interpretation?
Mapping motivations – Converted FA results for four
Coursera MOOCs
Factor I II III VI
INTSE
Experience an online course For fun and challenge
Relevant to academic
research
Relevant to job
Meet new people Personal growth&enrichment1,2
Relevant to academic
study
Earn a certificate
To improve my English skills General interest in topic For career change
Take with colleagues/friends
Course offered by prestigious
University/Professor,3
P2P
Experience an online course To improve my English skills Relevant to academic
study
For fun and challenge
Meet new people Relevant to academic research For career change Personal
growth&enrichment
Take with colleagues/friends Relevant to job General interest in topic
Course offered by prestigious
University/Professor
PMED
Relevant to academic study Meet new people
Relevant to job
Course offered by prestigious
University/Professor
For career change General interest in topic
HL
Experience an online course For career change General interest in topic
Meet new people
As an opportunity to think about
improving my career prosepects
Personal
growth&enrichment
To improve my English skills
To give me an opportunity to
explore an area I don't know about
Relevant to job
Relevant to academic research Earn a certificate
Relevant to academic study For fun and challenge
Earn a certificate
KMO = 0.72, total variance explained 64.7%
Three factors emerged -
1. Career development
2. Exploration of the learning
environment
3. Improving knowledge
‘I’m taking the course because…’ pre-course survey
n=5071
KMO = 0.82, total variance explained 64.36%
*henghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s
Benefiting from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-
benefiting-from-moocs-and-why
Two factors emerged -
1. Education seekers*
2. Career builders*
Were learners 'expectations met (Q9)
(n= 678)
Rotated Component Matrixa
Component
1 2
Q9_My overall expectations of the
course
.873 .059
Q9_Adding a fresh perspective to my
current role
.783 .370
Q9_Learning about the subject .776 .142
Q9_Learning more flexibly around my
other commitments
.657 .397
Q9_Supporting current or future
studies
.641 .510
Q9_Interacting with other people .448 .279
Q9_Supporting my university
application
.173 .856
Q9_Finding out more about the
University running the course
.175 .822
Q9_Improving my career prospects .523 .610
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.
a. Rotation converged in 3 iterations.
Explanation/definition of factors – motivation &
expectations
Courser Factors
1. Experience a new form of
education
2. Emphasize lifelong learning
3. To develop academic knowledge
4. To advance career prospects
FL Factors
1. Career development
2. Exploration of the learning
environment
3. Improving knowledge
Engagement – learners’ visits vs. educators’
engagement (Crosstab of reported engagement vs perceived level of engagement)
% as per column, indicates the percentage distribution of a particular rating point
Satisfaction with the experience
• Overall satisfaction with aspects of the course
• Perceived experience
• Value of the experience
Satisfaction of learning on FL
Post-course surveys of four MOOCs (n = 678)
% as per column, indicates the percentage distribution of a particular rating point
Satisfaction of course design and content
Post-course surveys of four MOOCs (n = 678)
% as per row, indicates the percentage distribution over the Likert Scale in each statement.
Sentiment analysis by step type (i.e. tool)
What are the aspect influencing the overall satisfaction rating?
Questions Coefficient* Sig’
Q3_Course orientation materials (i.e. course map video) 0.531 0.01
Q9_Learning about the subject 0.433 0.01
Q9_My overall expectations of the course 0.433 0.01
Q18_likely recommend 0.411 0.05
Q1_Doing quizzes and tests 0.411 0.05
Q9_Adding a fresh perspective to my current role 0.404 0.05
Q3_Written transcripts of all material 0.397 0.05
Q3_Use of animations in videos 0.361 0.05
Q1_Watching videos 0.343 0.05
Step 1. Correlation of questions with the overall
experience question (Q17)
Pearson correlation was chosen in the model.
What are the aspect influencing the overall satisfaction
rating? (cont`d)
Step 2. Linear regression (n=195)
Questions
Standardized
coefficient
significance
Q18_likely recommend 0.351 0.000
Q9_My overall expectations of the course 0.335 0.000
Q3_Course orientation materials (i.e. course map
video) 0.199 0.003
Q9_Learning about the subject 0.167 0.013
Compare with the four Coursera MOOCs regression model
results
Aspects
(Question Block
INTSE coefficient Sig'
Course
experience
Overall, this course met my expectations 0.54 0.00
LTTO
Course
experience
Overall this course met my expectations 0.57 0.00
Satisfaction Provided an opportunity for fun and challenge 0.10 0.03
P2P
Course
experience
Overall I have improved the knowledge/skills I will need 0.63 0.00
The course encouraged my interest in this field of study 0.61 0.01
PMED
Confidence and
strategies
I am confident that I could deal efficiently with unexpected events. 0.87 0.00
I can remain calm when facing difficulties because I can rely on my coping
abilities.
0.62 0.01
Course
experience
Depth of content 0.67 0.03
Pearson correlation was chosen in the model.
Discussions
• Does the SE evaluation model for formal education fit?
• What are the measures we can use to evaluate the experiences of MOOCs learners?
• What means do we have to effectively collect evidence?
• What evidence have we got to better understand how and why people learn, and hence to
evaluate their learning experience?
Images source: CALRG (Computers and Learning Research Group Annual Conference 2016,
Open University, UK https://twitter.com/CALRGatOU

More Related Content

What's hot

Achieve poll on college readiness
Achieve poll on college readinessAchieve poll on college readiness
Achieve poll on college readiness
RYHT
 
Katho New Media Syllabus Dec09 Ana Adi
Katho New Media Syllabus Dec09 Ana AdiKatho New Media Syllabus Dec09 Ana Adi
Katho New Media Syllabus Dec09 Ana Adi
Ana ADI
 
Elem edTPA Training ECU
Elem edTPA Training ECUElem edTPA Training ECU
Elem edTPA Training ECU
Liz Fogarty
 
PSPA_2520_Syllabus
PSPA_2520_SyllabusPSPA_2520_Syllabus
PSPA_2520_Syllabus
Ryan Griffith
 
FIPSE Findings
FIPSE FindingsFIPSE Findings
FIPSE Findings
CPEDInitiative
 
FIPSE Findings
FIPSE FindingsFIPSE Findings
FIPSE Findings
CPEDInitiative
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
Sarah Land
 
Ucdtlm0030
Ucdtlm0030Ucdtlm0030
Ucdtlm0030
Maricel Benavente
 
Slo administive training.jan2014
Slo administive training.jan2014Slo administive training.jan2014
Slo administive training.jan2014
Cissy Mecca
 
Academic Skills and Student Success in PSE
Academic Skills and Student Success in PSEAcademic Skills and Student Success in PSE
Academic Skills and Student Success in PSE
pdietsche
 
Developmental evaluations for institutional impact
Developmental evaluations for institutional impactDevelopmental evaluations for institutional impact
Developmental evaluations for institutional impact
Rhona Sharpe
 
Notes: What Do You Do When You Do What You Do with Student Ratings?
Notes: What Do You Do When You Do What You Do with Student Ratings?Notes: What Do You Do When You Do What You Do with Student Ratings?
Notes: What Do You Do When You Do What You Do with Student Ratings?
Thomas J. Tobin
 
Slides For Longbridge V2
Slides For Longbridge V2Slides For Longbridge V2
Slides For Longbridge V2
Becka Colley-Foster
 
Towards a cross boundary collaborative open learning framework latest
Towards a cross boundary collaborative open learning framework latestTowards a cross boundary collaborative open learning framework latest
Towards a cross boundary collaborative open learning framework latest
Chrissi Nerantzi
 
Teacher Connect Session 4
Teacher Connect Session 4Teacher Connect Session 4
Teacher Connect Session 4
Sharon Seslija
 
What makes good feedback good? - Prof. Margaret Price
What makes good feedback good? - Prof. Margaret PriceWhat makes good feedback good? - Prof. Margaret Price
What makes good feedback good? - Prof. Margaret Price
mdxaltc
 
Creating Our Own Pathways
Creating Our Own PathwaysCreating Our Own Pathways
Creating Our Own Pathways
CISA-GMU
 
Authentic assessment: a case study
Authentic assessment: a case studyAuthentic assessment: a case study
Authentic assessment: a case study
Angelica Risquez
 
Forpd V Zygouris Coe Sloan C 2009
Forpd V Zygouris Coe Sloan C 2009Forpd V Zygouris Coe Sloan C 2009
Forpd V Zygouris Coe Sloan C 2009
Vicky Zygouris-Coe
 
Blended by Design: Classroom Assessment Techniques & Rubrics
Blended by Design: Classroom Assessment Techniques & RubricsBlended by Design: Classroom Assessment Techniques & Rubrics
Blended by Design: Classroom Assessment Techniques & Rubrics
EDUCAUSE
 

What's hot (20)

Achieve poll on college readiness
Achieve poll on college readinessAchieve poll on college readiness
Achieve poll on college readiness
 
Katho New Media Syllabus Dec09 Ana Adi
Katho New Media Syllabus Dec09 Ana AdiKatho New Media Syllabus Dec09 Ana Adi
Katho New Media Syllabus Dec09 Ana Adi
 
Elem edTPA Training ECU
Elem edTPA Training ECUElem edTPA Training ECU
Elem edTPA Training ECU
 
PSPA_2520_Syllabus
PSPA_2520_SyllabusPSPA_2520_Syllabus
PSPA_2520_Syllabus
 
FIPSE Findings
FIPSE FindingsFIPSE Findings
FIPSE Findings
 
FIPSE Findings
FIPSE FindingsFIPSE Findings
FIPSE Findings
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
 
Ucdtlm0030
Ucdtlm0030Ucdtlm0030
Ucdtlm0030
 
Slo administive training.jan2014
Slo administive training.jan2014Slo administive training.jan2014
Slo administive training.jan2014
 
Academic Skills and Student Success in PSE
Academic Skills and Student Success in PSEAcademic Skills and Student Success in PSE
Academic Skills and Student Success in PSE
 
Developmental evaluations for institutional impact
Developmental evaluations for institutional impactDevelopmental evaluations for institutional impact
Developmental evaluations for institutional impact
 
Notes: What Do You Do When You Do What You Do with Student Ratings?
Notes: What Do You Do When You Do What You Do with Student Ratings?Notes: What Do You Do When You Do What You Do with Student Ratings?
Notes: What Do You Do When You Do What You Do with Student Ratings?
 
Slides For Longbridge V2
Slides For Longbridge V2Slides For Longbridge V2
Slides For Longbridge V2
 
Towards a cross boundary collaborative open learning framework latest
Towards a cross boundary collaborative open learning framework latestTowards a cross boundary collaborative open learning framework latest
Towards a cross boundary collaborative open learning framework latest
 
Teacher Connect Session 4
Teacher Connect Session 4Teacher Connect Session 4
Teacher Connect Session 4
 
What makes good feedback good? - Prof. Margaret Price
What makes good feedback good? - Prof. Margaret PriceWhat makes good feedback good? - Prof. Margaret Price
What makes good feedback good? - Prof. Margaret Price
 
Creating Our Own Pathways
Creating Our Own PathwaysCreating Our Own Pathways
Creating Our Own Pathways
 
Authentic assessment: a case study
Authentic assessment: a case studyAuthentic assessment: a case study
Authentic assessment: a case study
 
Forpd V Zygouris Coe Sloan C 2009
Forpd V Zygouris Coe Sloan C 2009Forpd V Zygouris Coe Sloan C 2009
Forpd V Zygouris Coe Sloan C 2009
 
Blended by Design: Classroom Assessment Techniques & Rubrics
Blended by Design: Classroom Assessment Techniques & RubricsBlended by Design: Classroom Assessment Techniques & Rubrics
Blended by Design: Classroom Assessment Techniques & Rubrics
 

Viewers also liked

Presentation slides v2
Presentation slides v2Presentation slides v2
Presentation slides v2
Catherine Zhao
 
Wsu guest lecture_2015_czlv_v3
Wsu guest lecture_2015_czlv_v3Wsu guest lecture_2015_czlv_v3
Wsu guest lecture_2015_czlv_v3
Catherine Zhao
 
Effective student learning through goal setting v2
Effective student learning through goal setting v2Effective student learning through goal setting v2
Effective student learning through goal setting v2
Catherine Zhao
 
Myers MSN resume 2016
Myers MSN resume 2016Myers MSN resume 2016
Myers MSN resume 2016
Latoya Myers
 
HR Business Partner II
HR Business Partner IIHR Business Partner II
HR Business Partner II
Belinda Teller
 
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
mellenie runny
 
Ngwaqa Thabani
Ngwaqa ThabaniNgwaqa Thabani
Ngwaqa Thabani
Thabani Ngwaqa
 
Consolidated Story Book Linkedin
Consolidated Story Book LinkedinConsolidated Story Book Linkedin
Consolidated Story Book Linkedin
Avnee Thakrar
 

Viewers also liked (8)

Presentation slides v2
Presentation slides v2Presentation slides v2
Presentation slides v2
 
Wsu guest lecture_2015_czlv_v3
Wsu guest lecture_2015_czlv_v3Wsu guest lecture_2015_czlv_v3
Wsu guest lecture_2015_czlv_v3
 
Effective student learning through goal setting v2
Effective student learning through goal setting v2Effective student learning through goal setting v2
Effective student learning through goal setting v2
 
Myers MSN resume 2016
Myers MSN resume 2016Myers MSN resume 2016
Myers MSN resume 2016
 
HR Business Partner II
HR Business Partner IIHR Business Partner II
HR Business Partner II
 
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
Hari kantin sekolah menengah saint francis xavier,.pptx (assigment)
 
Ngwaqa Thabani
Ngwaqa ThabaniNgwaqa Thabani
Ngwaqa Thabani
 
Consolidated Story Book Linkedin
Consolidated Story Book LinkedinConsolidated Story Book Linkedin
Consolidated Story Book Linkedin
 

Similar to Cz la eds-jun-21_v3

Course design and delivery: guidance and tips for impactful training
Course design and delivery: guidance and tips for impactful trainingCourse design and delivery: guidance and tips for impactful training
Course design and delivery: guidance and tips for impactful training
CORBEL
 
Self directed learning in future learn courses using the Bouchard framework
Self directed learning in future learn courses using the Bouchard frameworkSelf directed learning in future learn courses using the Bouchard framework
Self directed learning in future learn courses using the Bouchard framework
Inge de Waard
 
Sb iep module-3
Sb iep module-3Sb iep module-3
Sb iep module-3
Elaine Karp
 
The Art of Designing Effective Teaching and Significant Learning
The Art of Designing Effective Teaching and Significant LearningThe Art of Designing Effective Teaching and Significant Learning
The Art of Designing Effective Teaching and Significant Learning
D2L Barry
 
Tpack planning
Tpack planningTpack planning
Tpack planning
Carla Piper
 
Education Abroad and Formative Outcomes Assessment: What Professors Think The...
Education Abroad and Formative Outcomes Assessment: What Professors Think The...Education Abroad and Formative Outcomes Assessment: What Professors Think The...
Education Abroad and Formative Outcomes Assessment: What Professors Think The...
CIEE
 
Investing in Paraeducator Capacity
Investing in Paraeducator CapacityInvesting in Paraeducator Capacity
Investing in Paraeducator Capacity
National Resource Center for Paraprofessionals
 
Secondary Session 4 A 09
Secondary Session 4 A 09Secondary Session 4 A 09
Secondary Session 4 A 09
Sharon Seslija
 
Blended By Design: Student Readiness, Student Crisis Points, and Student Teams
Blended By Design: Student Readiness, Student Crisis Points, and Student TeamsBlended By Design: Student Readiness, Student Crisis Points, and Student Teams
Blended By Design: Student Readiness, Student Crisis Points, and Student Teams
EDUCAUSE
 
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
Centre for Distance Education
 
1410 young
1410 young1410 young
Exceptional learning needs
Exceptional learning needsExceptional learning needs
Exceptional learning needs
Cynthia Li
 
The Mindful Instruction Librarian and the "One-Shot"
The Mindful Instruction Librarian and the "One-Shot"The Mindful Instruction Librarian and the "One-Shot"
The Mindful Instruction Librarian and the "One-Shot"
Meredith Farkas
 
Conducting Research on Blended and Online Education, Workshop
Conducting Research on Blended and Online Education, WorkshopConducting Research on Blended and Online Education, Workshop
Conducting Research on Blended and Online Education, Workshop
Tanya Joosten
 
Raising achievement and ensuring success for students of History
Raising achievement and ensuring success for students of HistoryRaising achievement and ensuring success for students of History
Raising achievement and ensuring success for students of History
wigifer
 
High School Guidance Presentation - 06/23/15
High School Guidance Presentation - 06/23/15High School Guidance Presentation - 06/23/15
High School Guidance Presentation - 06/23/15
Franklin Matters
 
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
Iwan Syahril
 
Building Foundations for Para-professionalism
Building Foundations for Para-professionalismBuilding Foundations for Para-professionalism
Building Foundations for Para-professionalism
National Resource Center for Paraprofessionals
 
Guatemala science course redesign 2 11-01-11
Guatemala science course redesign 2 11-01-11Guatemala science course redesign 2 11-01-11
Guatemala science course redesign 2 11-01-11
marorussell
 
Blended Course Design
Blended Course DesignBlended Course Design
Blended Course Design
Jason Rhode
 

Similar to Cz la eds-jun-21_v3 (20)

Course design and delivery: guidance and tips for impactful training
Course design and delivery: guidance and tips for impactful trainingCourse design and delivery: guidance and tips for impactful training
Course design and delivery: guidance and tips for impactful training
 
Self directed learning in future learn courses using the Bouchard framework
Self directed learning in future learn courses using the Bouchard frameworkSelf directed learning in future learn courses using the Bouchard framework
Self directed learning in future learn courses using the Bouchard framework
 
Sb iep module-3
Sb iep module-3Sb iep module-3
Sb iep module-3
 
The Art of Designing Effective Teaching and Significant Learning
The Art of Designing Effective Teaching and Significant LearningThe Art of Designing Effective Teaching and Significant Learning
The Art of Designing Effective Teaching and Significant Learning
 
Tpack planning
Tpack planningTpack planning
Tpack planning
 
Education Abroad and Formative Outcomes Assessment: What Professors Think The...
Education Abroad and Formative Outcomes Assessment: What Professors Think The...Education Abroad and Formative Outcomes Assessment: What Professors Think The...
Education Abroad and Formative Outcomes Assessment: What Professors Think The...
 
Investing in Paraeducator Capacity
Investing in Paraeducator CapacityInvesting in Paraeducator Capacity
Investing in Paraeducator Capacity
 
Secondary Session 4 A 09
Secondary Session 4 A 09Secondary Session 4 A 09
Secondary Session 4 A 09
 
Blended By Design: Student Readiness, Student Crisis Points, and Student Teams
Blended By Design: Student Readiness, Student Crisis Points, and Student TeamsBlended By Design: Student Readiness, Student Crisis Points, and Student Teams
Blended By Design: Student Readiness, Student Crisis Points, and Student Teams
 
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
CDE Conference 09/02/2009. W Morrison. Assessment and evaluation in UG Laws: ...
 
1410 young
1410 young1410 young
1410 young
 
Exceptional learning needs
Exceptional learning needsExceptional learning needs
Exceptional learning needs
 
The Mindful Instruction Librarian and the "One-Shot"
The Mindful Instruction Librarian and the "One-Shot"The Mindful Instruction Librarian and the "One-Shot"
The Mindful Instruction Librarian and the "One-Shot"
 
Conducting Research on Blended and Online Education, Workshop
Conducting Research on Blended and Online Education, WorkshopConducting Research on Blended and Online Education, Workshop
Conducting Research on Blended and Online Education, Workshop
 
Raising achievement and ensuring success for students of History
Raising achievement and ensuring success for students of HistoryRaising achievement and ensuring success for students of History
Raising achievement and ensuring success for students of History
 
High School Guidance Presentation - 06/23/15
High School Guidance Presentation - 06/23/15High School Guidance Presentation - 06/23/15
High School Guidance Presentation - 06/23/15
 
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
A workshop on writing "Rencana Pembelajaran Semester" (RPS) and "Satuan Acara...
 
Building Foundations for Para-professionalism
Building Foundations for Para-professionalismBuilding Foundations for Para-professionalism
Building Foundations for Para-professionalism
 
Guatemala science course redesign 2 11-01-11
Guatemala science course redesign 2 11-01-11Guatemala science course redesign 2 11-01-11
Guatemala science course redesign 2 11-01-11
 
Blended Course Design
Blended Course DesignBlended Course Design
Blended Course Design
 

Recently uploaded

V2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docxV2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docx
302491
 
3. Maturity_indices_of_fruits_and_vegetable.pptx
3. Maturity_indices_of_fruits_and_vegetable.pptx3. Maturity_indices_of_fruits_and_vegetable.pptx
3. Maturity_indices_of_fruits_and_vegetable.pptx
UmeshTimilsina1
 
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
Codeavour International
 
Demonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 SlidesDemonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 Slides
Celine George
 
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
ALBERTHISOLER1
 
7. Post Harvest Entomology and their control.pptx
7. Post Harvest Entomology and their control.pptx7. Post Harvest Entomology and their control.pptx
7. Post Harvest Entomology and their control.pptx
UmeshTimilsina1
 
11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
UmeshTimilsina1
 
Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17
Celine George
 
FINAL MATATAG Science CG 2023 Grades 3-10.pdf
FINAL MATATAG Science CG 2023 Grades 3-10.pdfFINAL MATATAG Science CG 2023 Grades 3-10.pdf
FINAL MATATAG Science CG 2023 Grades 3-10.pdf
maritescanete2
 
6. Physiological Disorder of fruits and vegetables.pptx
6. Physiological Disorder of fruits and vegetables.pptx6. Physiological Disorder of fruits and vegetables.pptx
6. Physiological Disorder of fruits and vegetables.pptx
UmeshTimilsina1
 
SQL Server Interview Questions PDF By ScholarHat
SQL Server Interview Questions PDF By ScholarHatSQL Server Interview Questions PDF By ScholarHat
SQL Server Interview Questions PDF By ScholarHat
Scholarhat
 
Lecture Notes Unit4 Chapter13 users , roles and privileges
Lecture Notes Unit4 Chapter13 users , roles and privilegesLecture Notes Unit4 Chapter13 users , roles and privileges
Lecture Notes Unit4 Chapter13 users , roles and privileges
Murugan146644
 
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
mansk2
 
Dot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHatDot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHat
Scholarhat
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Alvaro Barbosa
 
2024 Winter SWAYAM NPTEL & A Student.pptx
2024 Winter SWAYAM NPTEL & A Student.pptx2024 Winter SWAYAM NPTEL & A Student.pptx
2024 Winter SWAYAM NPTEL & A Student.pptx
Utsav Yagnik
 
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
MANIVALANSR
 
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdfOpen Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Jessica Zairo
 
5. Postharvest deterioration of fruits and vegetables.pptx
5. Postharvest deterioration of fruits and vegetables.pptx5. Postharvest deterioration of fruits and vegetables.pptx
5. Postharvest deterioration of fruits and vegetables.pptx
UmeshTimilsina1
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
Scholarhat
 

Recently uploaded (20)

V2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docxV2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docx
 
3. Maturity_indices_of_fruits_and_vegetable.pptx
3. Maturity_indices_of_fruits_and_vegetable.pptx3. Maturity_indices_of_fruits_and_vegetable.pptx
3. Maturity_indices_of_fruits_and_vegetable.pptx
 
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
Codeavour 5.0 International Impact Report - The Biggest International AI, Cod...
 
Demonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 SlidesDemonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 Slides
 
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
 
7. Post Harvest Entomology and their control.pptx
7. Post Harvest Entomology and their control.pptx7. Post Harvest Entomology and their control.pptx
7. Post Harvest Entomology and their control.pptx
 
11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
 
Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17
 
FINAL MATATAG Science CG 2023 Grades 3-10.pdf
FINAL MATATAG Science CG 2023 Grades 3-10.pdfFINAL MATATAG Science CG 2023 Grades 3-10.pdf
FINAL MATATAG Science CG 2023 Grades 3-10.pdf
 
6. Physiological Disorder of fruits and vegetables.pptx
6. Physiological Disorder of fruits and vegetables.pptx6. Physiological Disorder of fruits and vegetables.pptx
6. Physiological Disorder of fruits and vegetables.pptx
 
SQL Server Interview Questions PDF By ScholarHat
SQL Server Interview Questions PDF By ScholarHatSQL Server Interview Questions PDF By ScholarHat
SQL Server Interview Questions PDF By ScholarHat
 
Lecture Notes Unit4 Chapter13 users , roles and privileges
Lecture Notes Unit4 Chapter13 users , roles and privilegesLecture Notes Unit4 Chapter13 users , roles and privileges
Lecture Notes Unit4 Chapter13 users , roles and privileges
 
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
 
Dot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHatDot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHat
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
 
2024 Winter SWAYAM NPTEL & A Student.pptx
2024 Winter SWAYAM NPTEL & A Student.pptx2024 Winter SWAYAM NPTEL & A Student.pptx
2024 Winter SWAYAM NPTEL & A Student.pptx
 
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
 
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdfOpen Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdf
 
5. Postharvest deterioration of fruits and vegetables.pptx
5. Postharvest deterioration of fruits and vegetables.pptx5. Postharvest deterioration of fruits and vegetables.pptx
5. Postharvest deterioration of fruits and vegetables.pptx
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
 

Cz la eds-jun-21_v3

  • 1. Learning Analytics & Data Science Research Group – 21 June 2016 Different Sides of MOOCs – Comparing Perceptions and Experiences across Platforms Catherine Zhao & Lorenzo Vigentini
  • 2. Outline • Recap of the ‘flipped’ paper • Key features of Coursera & FutureLearn (FL) • In Coursera & FL 1. Various feedback gathering means 2. MOOCs Learners’ demographic profiles 3. Elements influencing MOOCs Learners’ experiences 4. The overall satisfaction with experience
  • 3. Recap of ‘Evaluating the student experience in MOOCs’ http://bit.ly/flipMoocExp
  • 4. The context of the paper - four Coursera MOOCs http://bit.ly/flipMoocExp INTSE LTTO P2P PMED Disciplines Engineering Education Science Medicine Duration (weeks) 9 6 9 8 Target Group Engineers Teachers High school students/ teachers General public Course Design All-at-once All-at-once Sequential Semi-sequential Course Delivery All-at-once Staggered Staggered Staggered Videos 110 224 98 58 Quizzes 10 22 42 2 Assignments 7 3 2 2 Forums 54 105 63 64 Registrants (Active learners*) 32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%) *Active learner: those who appear at least once during the course
  • 5. Highlight 1- Design drives learning to some extent Example– engagement with quizzes INTSE P2P PersMed LTTO
  • 6. Highlight 2. in-context works better to collect data 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 INTSE Views of lecture videos 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 P2P Views of lecture videos
  • 7. Highlight 3-Model of evaluating mainstream SE could also be applied in MOOCs Course Enablers are the key features of the educational platform, the course designs and mode of teaching to support learning. Precursors are students’ prior academic and professional experience, motivations, aspirations and expectations of learning. Engagement is the totality of learners’ interactions with the platform, peers, and the course. The Worth of Experience is what learners take away and project to their future, evaluating the learning experience, their satisfaction with the course, the perceived utility and effectiveness of the course for their learning, and/or for the development of knowledge and skills. Conceptual building blocks characterizing the student experience
  • 8. – Instructivist approach (Vigentini & Clayphan, 2015) – Mastery learning (Coursera principles*) – Networked/connected learning (FL principles**) – Openness (FL principles) Key principles of Coursera & FL platforms Vigentini, L & Clayphan, A. (2015) Pacing through MOOCs: course design or teaching effect? International conference of educational data mining, Madrid, Spain *Coursera principles https://www.coursera.org/about/ **FL principles https://www.futurelearn.com/about/our-principles
  • 9. Platform content assessment activity Discussion forum Coursera Y Y Y Y FL Y Kinda no Y Errrr… Key feature of MOOCs on Coursera & FL
  • 10. Coursera’s standard survey instruments + ‘In-house’ surveys by offering institutions Throughout offering period FL’s standard pre- & post- course surveys Stand-alone Throughout offering period Feedback gathering in Coursera & FL
  • 11. Feedback gathering in Coursera & FL (cont`d) *Coursera standard survey **Coursera no longer deployed ***Coursera no long supported for evaluation purposes Deployment period Evaluative Instruments in Coursera Evaluative Instruments in FL Prior to course commencement Demographic survey* Pre-course Survey Pre-course survey End of course Post-course survey Post-course Survey During the course Quick evaluation (Video) Quick evaluation (Module) Quick Question survey** In-video surveys (IVS)***
  • 12. The pre- & post-course surveys in Coursera Areas Pre-course survey question key words Area Post-course survey question key words Motivation Reasons of joining Satisfaction Course Experience expectation expected outcomes Satisfaction of Personal gains intended weekly hours Engagement Actual Effort Perception of ‘Completion' Proportion of Completed Course Activities Preventing Factors for Course Completion Future aspiration (Intend to) Apply Course Topics Learning preference Consider the way you prefer to learn Considering Credit-Bearing Courses Demographics Prior professional experience Evaluation of course design Peer Assessment Familiarisation with Topics (specific) Course Features Qualification Self-efficacy Confidence and strategies of problem solving Background-Gender, Ethnicity English Proficiency Prior academic experience Geographic Location Personal goals What goals have you set for yourself in this course
  • 13. Mapping of items between pre-course surveys in FLPre-course survey version I Pre-course survey version II hope to get out of this course? I am taking this course because… Additional thoughts about your expectations of FutureLearn. Additional thoughts about your expectations of FutureLearn. Have you taken a course delivered mostly or fully online before, including Massive Open Online Courses (MOOCs)? Have you taken a course delivered mostly or fully online before, including Massive Open Online Courses (MOOCs)? What sort of online course have you taken? What sort of online learning have you participated in? How did you find out about this course? How did you find out about this course? Country Which of the following related roles have you held in the past year?Age group Gender Employment status Industry of employment? Qualification Do you have a disability, long-term health condition, mental health condition, specific learning difficulty (such as dyslexia), or other physical or mental impairment? What previous experience, if any, do you have in this subject area? How you would like to learn on FL Where do you expect to do the course Subject areas of interested
  • 14. The pre-course surveys in FL Areas Pre-course survey version I Areas Pre-course survey version II Expectations hope to get out of this course? I am taking this course because… Additional thoughts about your expectations of FutureLearn. Additional thoughts about your expectations of FutureLearn. Prior exposure to online learning Have you taken a course delivered mostly or fully online before, including Massive Open Online Courses (MOOCs)? Have you taken a course delivered mostly or fully online before, including Massive Open Online Courses (MOOCs)? What sort of online course have you taken? What sort of online learning have you participated in? Marketing and communication How did you find out about this course? How did you find out about this course? Demographics Country Which of the following related roles have you held in the past year?Age group Gender Employment status Industry of employment? Qualification Health conditions Do you have a disability, long-term health condition, mental health condition, specific learning difficulty (such as dyslexia), or other physical or mental impairment? Prior experience What previous experience, if any, do you have in this subject area? Preferences for learning How you would like to learn on FL Where do you expect to do the course subject of interest Subject areas of interested
  • 15. The post-course surveys in FL Areas Post-course survey question key words Learning preferences How you felt about learning on FutureLearn. Where did you do the course? Aspects of technology How easy was it to find your way around the site? Expectation How did the course difficulty meet with your expectations? To what extent did FutureLearn meet your expectations Engagement Roughly how often did you visit the course? How did you feel about the length of the course? Evaluation of course design How you felt about the course design and content. To what extent did you find the educator(s) engaging? Your favourite part of the course Your least favourite part of the course How could the course be improved? Value of a statement How likely are you to purchase a Statement of Participation for this course? Why are you interested in a Statement of Participation? Please let us know why you are unlikely to purchase a Statement of Participation What would improve a Statement of Participation? (Satisfaction) of overall experience How would you rate your overall experience of the course? Future aspiration How will you pursue your interest in the subject now that the course is complete? What would you like to study next? Marketing and communication How likely would you be to recommend FL to a friend?
  • 16. Related study focusing on SDL Areas Key categories Concept Personal learning identity Time/planning Leisure/casual learning Health as a driver 1. Daily life elements affecting the learning experience Technology as driver (good/bad) Learning mobility 2. Technical aspect affecting the learning experience Balance known & new Changing learning pace 3. Actions undertaken to adapt learning Various items deployed across questions 4. Perception of individual & collaborative learning An experience of self-directed learning (Knowles 1975, Merriam 2001, de Waard et al. 2015) de Waard, Inge; Kukulska-Hulme, Agnes and Sharples, Mike (2015). Self-Directed Learning in Trial FutureLearn courses. In: Proceedings Papers, EMOOCS, pp. 234–243.
  • 17. FL Week/Module evaluation (for Adaptive tutorials) Randomised/balanced item presentation (to reduce number of questions) in 5 different areas: • Reasons to enroll (ranking of 3 items) • Engagement preferences (2 items) • Satisfaction (2 overall + 2 items) • Rating of adaptive learning (4 items) • Successful completion preference (4 items)
  • 19. Post-course survey samples Coursera MOOC # active learners Post-course survey sample response rate Margin of error INTSE 19708 638 3.24% 3.82% LTTO 18024 313 1.74% 5.49% P2P 10576 126 1.19% 8.68% HL 4673 71 1.52% 11.54% FL MOOC Total registrants # post-course survey responses Post-course survey response rate Margin of Error WWI lessons legacy 9747 396 4.06% 4.82% Military ethics 6218 102 1.64% 9.62% Remaking nature 7907 127 1.61% 9.63% Maths linear quadratic 8757 49 0.56% 13.96%
  • 20. What’s the difference in feedback gathering? (Cont`d) *Coursera standard survey **Coursera no longer deployed ***Coursera no long supported for evaluation purposes Deployment period Evaluative Instruments in Coursera Evaluative Instruments in FL Prior to course commencement Demographic survey* Pre-course Survey Pre-course survey End of course Post-course survey Post-course Survey During the course Quick evaluation (Video) Quick evaluation (Module) Adaptive tutorial evaluation Quick Question survey** In-video surveys (IVS)***
  • 21. The diversity of learners across 4 FL MOOCs Demographic INTSE LTTO P2P PMED Typical Learners profile in a Cousera MOOC* Ave. Age (SD) 34.37 (11.37) 39.9 (12.41) 35.9 (15) 37.35 (14) 35 Gender ratio (M-F) 4.0-1.0 1-1.2 2.6-1 1-1.2 1.7-1.0 Full-time Employed 57% 63% 59% 59% - Full-time Studying 41% 30% 39% 38% 27% Bachelors and Above 74% 82% 63% 64% 70% Prior exposure to online learning - - - - - Military ethics Math_algebra World War I Remaking nature Typical Learners profile in a FL MOOC Ave. Age (SD) 36-45 26-35 56-65 36-45 Gender ratio (M-F) 1.5-1.0 1.0-1.0 1.2-1.0 1-2.23 Full-time Employed 7% 4% 18% 5% Full-time Studying 3% 2% 3% 2% Bachelors and Above 14% 11% 40% 13% Prior exposure to online learning 67.85% 61.83% 75.64% 48.28%
  • 22. Preferences for learning - Preferred ways to engage with the course, content and others - Where/how accessing the course - Learning strategies?
  • 23. Distribution of responses for preferred ways to learn (4 MOOCs, N= 5071) % as per column, indicates the percentage distribution of a particular rating point i.e. of all ratings over ‘strongly like’ 25% on ‘preferred to learn by reading text’.
  • 24. Distribution of responses for preferred ways to learn (split by previous online experience) % as per column, split by previous online experience, indicates the percentage distribution of a particular rating point i.e. of all ratings over ‘strongly like’ 17.12% with prior exposure to online learning on ‘preferred to learn by reading text’.
  • 25. Where did learners prefer to learn Math_algebra military ethics remaking nature World war I lessons legacy At work 9.93% 13.25% 11.68% 8.47% At school / college / university 6.68% 6.53% 5.47% 2.64% During my commute 2.04% 4.09% 2.86% 2.46% At home at a desk 47.12% 39.55% 41.58% 48.72% At home on the sofa 25.70% 25.51% 23.53% 29.45% In a public place with internet access 5.84% 7.32% 9.89% 5.03% Out and about 2.69% 3.76% 4.98% 3.24% N=5071 % as per column, indicates the percentage distribution of a particular rating point i.e. of all math_algebra responses, 47.12% chose ‘at home at a desk’.
  • 26. Motivations and Expectations • What is expected from the course • Reasons for taking the course
  • 27. Pre Course – Expectation distribution (4 MOOCs) % as per column, indicates the percentage distribution per MOOC. Note: people could choose as many options as they like.
  • 28. Pre Course – Motivation distribution (IFL/TEE) % as per row, indicates the percentage distribution over the Likert Scale in each statement.
  • 29. What are the emerging factors in motivation? • Are the wording of the questions effective? • Which questions are more closely correlated with each other? • Can we reduce the dimensions for interpretation?
  • 30. Mapping motivations – Converted FA results for four Coursera MOOCs Factor I II III VI INTSE Experience an online course For fun and challenge Relevant to academic research Relevant to job Meet new people Personal growth&enrichment1,2 Relevant to academic study Earn a certificate To improve my English skills General interest in topic For career change Take with colleagues/friends Course offered by prestigious University/Professor,3 P2P Experience an online course To improve my English skills Relevant to academic study For fun and challenge Meet new people Relevant to academic research For career change Personal growth&enrichment Take with colleagues/friends Relevant to job General interest in topic Course offered by prestigious University/Professor PMED Relevant to academic study Meet new people Relevant to job Course offered by prestigious University/Professor For career change General interest in topic HL Experience an online course For career change General interest in topic Meet new people As an opportunity to think about improving my career prosepects Personal growth&enrichment To improve my English skills To give me an opportunity to explore an area I don't know about Relevant to job Relevant to academic research Earn a certificate Relevant to academic study For fun and challenge Earn a certificate
  • 31. KMO = 0.72, total variance explained 64.7% Three factors emerged - 1. Career development 2. Exploration of the learning environment 3. Improving knowledge ‘I’m taking the course because…’ pre-course survey n=5071
  • 32. KMO = 0.82, total variance explained 64.36% *henghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos- benefiting-from-moocs-and-why Two factors emerged - 1. Education seekers* 2. Career builders* Were learners 'expectations met (Q9) (n= 678) Rotated Component Matrixa Component 1 2 Q9_My overall expectations of the course .873 .059 Q9_Adding a fresh perspective to my current role .783 .370 Q9_Learning about the subject .776 .142 Q9_Learning more flexibly around my other commitments .657 .397 Q9_Supporting current or future studies .641 .510 Q9_Interacting with other people .448 .279 Q9_Supporting my university application .173 .856 Q9_Finding out more about the University running the course .175 .822 Q9_Improving my career prospects .523 .610 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 3 iterations.
  • 33. Explanation/definition of factors – motivation & expectations Courser Factors 1. Experience a new form of education 2. Emphasize lifelong learning 3. To develop academic knowledge 4. To advance career prospects FL Factors 1. Career development 2. Exploration of the learning environment 3. Improving knowledge
  • 34. Engagement – learners’ visits vs. educators’ engagement (Crosstab of reported engagement vs perceived level of engagement) % as per column, indicates the percentage distribution of a particular rating point
  • 35. Satisfaction with the experience • Overall satisfaction with aspects of the course • Perceived experience • Value of the experience
  • 36. Satisfaction of learning on FL Post-course surveys of four MOOCs (n = 678) % as per column, indicates the percentage distribution of a particular rating point
  • 37. Satisfaction of course design and content Post-course surveys of four MOOCs (n = 678) % as per row, indicates the percentage distribution over the Likert Scale in each statement.
  • 38. Sentiment analysis by step type (i.e. tool)
  • 39. What are the aspect influencing the overall satisfaction rating? Questions Coefficient* Sig’ Q3_Course orientation materials (i.e. course map video) 0.531 0.01 Q9_Learning about the subject 0.433 0.01 Q9_My overall expectations of the course 0.433 0.01 Q18_likely recommend 0.411 0.05 Q1_Doing quizzes and tests 0.411 0.05 Q9_Adding a fresh perspective to my current role 0.404 0.05 Q3_Written transcripts of all material 0.397 0.05 Q3_Use of animations in videos 0.361 0.05 Q1_Watching videos 0.343 0.05 Step 1. Correlation of questions with the overall experience question (Q17) Pearson correlation was chosen in the model.
  • 40. What are the aspect influencing the overall satisfaction rating? (cont`d) Step 2. Linear regression (n=195) Questions Standardized coefficient significance Q18_likely recommend 0.351 0.000 Q9_My overall expectations of the course 0.335 0.000 Q3_Course orientation materials (i.e. course map video) 0.199 0.003 Q9_Learning about the subject 0.167 0.013
  • 41. Compare with the four Coursera MOOCs regression model results Aspects (Question Block INTSE coefficient Sig' Course experience Overall, this course met my expectations 0.54 0.00 LTTO Course experience Overall this course met my expectations 0.57 0.00 Satisfaction Provided an opportunity for fun and challenge 0.10 0.03 P2P Course experience Overall I have improved the knowledge/skills I will need 0.63 0.00 The course encouraged my interest in this field of study 0.61 0.01 PMED Confidence and strategies I am confident that I could deal efficiently with unexpected events. 0.87 0.00 I can remain calm when facing difficulties because I can rely on my coping abilities. 0.62 0.01 Course experience Depth of content 0.67 0.03 Pearson correlation was chosen in the model.
  • 42. Discussions • Does the SE evaluation model for formal education fit? • What are the measures we can use to evaluate the experiences of MOOCs learners? • What means do we have to effectively collect evidence? • What evidence have we got to better understand how and why people learn, and hence to evaluate their learning experience? Images source: CALRG (Computers and Learning Research Group Annual Conference 2016, Open University, UK https://twitter.com/CALRGatOU

Editor's Notes

  1. Overview: integrated approach requiring all ‘ingredients’ and coordination
  2. Randomised from pool, balanced based on number of responses so all options are roughly equally presented
  3. Table down Compare expectations across MOOCs