SlideShare a Scribd company logo
Evaluating the ‘Student’ Experience in
Massive Open Online Courses (MOOCs)
Catherine Zhao & Lorenzo Vigentini
Learning and Teaching Unit, UNSW Australia
In this presentation -
• Background – Previous work on Student learning
Experience & UNSW MOOCs
• Methods - the surveys for evaluation
• Findings -
o Surveys - fit-for-purpose?
o Learners – what’s it like to experience a MOOC?
• Current work & Challenges
A Model of the Student Learning Experience
Worth of
Experience
Future
Past
Motivation
Experience
Perception
Engagement
Aspirations
Expectations
Personal
development
‘match’ or ‘fit’ with
context/LE
Direct future
benefits
Vigentini, L., and Zhao, C. Revisiting end-of-semester evaluation: the questions, the instrument and the
process.,” in AAIR : turning silver into gold, Melbourne, 2014.
Background – Why UNSW joined the ‘bandwagon’
Background- What it takes to design and deliver a MOOC
Five known facts about MOOCs
Massive enrollments
Dramatic dropouts
Unconventional learners
(Relatively) limited design/delivery options
Engagement and the learner experience
The rise of ‘certificates’
Background - Four MOOCs we look at
A B C D
Disciplines Engineering Education Science Medicine
Duration (weeks) 9 6 9 8
Target Group Engineers Teachers
High school
students/ teachers
General public
Course Design All-at-once All-at-once Sequential Semi-sequential
Course Delivery All-at-once Staggered Staggered Staggered
Videos 110 224 98 58
Quizzes 10 22 42 2
Assignments 7 3 2 2
Forums 54 105 63 64
Registrants (Active
learners*)
32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%)
*Active learner: those who appear at least once during the course
MOOCs – A = engineering, B = education, C = physics, D = medicine
Background- What & When Surveys
Evaluation Surveys Deployment A B C D
Demographic survey
Prior to course
commencement
   
Pre-course survey OR
First course activity
   
In-video surveys (IVS)
During
the
course

Quick evaluation (Video) 
Quick evaluation (Module) 
Quick Question survey  
Post-course survey End of course    
 Present in MOOC
MOOCs – A = engineering, B = education, C = physics, D = medicine
Methods - Overview of the Pre-Course survey questions
Common A B C D Course-specific A B C D
Reasons/Purpose of Joining    Background-Gender, Ethnicity 
Prior Professional
Experience/Knowledge
    English Proficiency 
Intended Effort    Prior MOOC Experience 
Familiarisation with Topics
 
Geographic Location 
Academic Qualification   
Contributing Elements of
‘Teaching Online’

Industry of Employment

 Defining ‘Completion' 
Preventing Factors for Course
Completion
 
Learning Preference 
Expected Outcomes
(course related)
  Achievement Goal Oriented 
 Present in the course
MOOCs – A = engineering, B = education, C = physics, D = medicine
Common A B C D Course-specific A B C D
Course Satisfaction 
   Proportion of Completed Course
Activities

Course Features     Evaluating Peer Assessment 
Actual Effort 

  (Probing) Self-Regulated Learning 
Desired Topics   
Satisfaction on Personal Development
 

(Intend to) Apply Course Topics 
  
Considering Credit-Bearing Courses
 
Methods- Overview of the Post-course survey questions
MOOCs – A = engineering, B = education, C = physics, D = medicine
Methods– Pre- & Post-Course Survey Sample Sizes
A B C D
# Active Learners 19708 18024 10576 4673
Required sample size* 2141 2119 1957 1587
Pre- & Post- Course survey
Pre-course survey sample 2684 4490 264 401
response rate 13.62% 24.91% 2.50% 8.58%
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
response rate 3.24% 1.74% 1.19% 1.52%
Margin of error 3.82% 5.49% 8.68% 11.54%
Coursera Demographic Survey
# of registrants 32928 28864 22487 13185
Demographic Survey Responses 2774 2873 2720 1466
Margin of error 1.20% 2.20% 1.52% 2.41%
*Required sample size is calculated under the condition of CL = 95%, and margin of error = 2%
Methods – Patterns of participation in Pre-& Post-
Course surveys
Counts of responses in the four MOOCs’ pre-course surveys (logarithmic scale) over 9 weeks.
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses A
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
B
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses
C
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
D
Pre Post
Methods- Collection tools
Course A IVS (In-video survey) Course C Quick Evaluation
MOOCs – A = engineering, B = education, C = physics, D = medicine
Research Questions
1) Are the tools used robust enough to characterize the
learners’ experience in MOOC?
2) Are there substantial differences in the modes of
gathering feedback?
3) Is it possible to identify key elements of the learners’
MOOC experiences?
4) What is the relevance of ‘teaching’ questions for the
online experience?
Results – Are the surveys robust?
• Accuracy
• A B C D
Required sample size (CI=95%,margin of error 2%) 2141 2119 1957 1587
Pre-course survey sample 2684 4490 264 401
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
Margin of error 3.82% 5.49% 8.68% 11.54%
Results – Are the surveys robust? (Cont’d)
A B C D
Motivation Block 0.78 0.86 0.65
Course experience Block 0.90 0.94 0.89 0.92
IVS items 0.82
Confidence 0.84
Satisfaction of personal gain 0.91
Satisfaction of course materials and
personal gain
0.92
(probing) self-regulated learning skills 0.90
• Reliability – Cronbach’s alpha
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Are the surveys robust? (Cont’d)
• For the motivation block that is <.7
(D) Reason items
Cronbach's Alpha if
Item Deleted
I am taking the course out of general interest, curiosity, or
enjoyment.
.722
I am interested in taking a course from this particular
institution.
.616
I am interested in learning from the expert researchers
and clinicians delivering the course.
.636
I want to connect with other students interested in this
topic.
.589
The course supports my current academic program. .565
The course is aligned with my current job responsibilities
or company's line-of-business.
.588
The skills gained from this course may be useful for
obtaining a new job.
.571
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Comparison of Evaluation Approaches
• The reasons for adopting different approaches
1. Technical constraints
2. Accommodating various requirements put
forward by academic leads
3. Test different tools/ways to evaluate
• Two key sets of approaches we are comparing -
1. First course activity in A & B vs. Pre-course
survey in C & D
2. In-Video Surveys (IVS) vs. Quick Evaluation (QE)
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Effectiveness of Pre-course survey
• ‘Embedded’ (= first course activity) is more effective
than ‘stand-alone’
• Higher response rate
• measures how much learners want to engage
A B C D
13.62% 24.91% 2.50% 8.58%
MOOCs – A = engineering, B = education, C = physics, D = medicine
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
A - IVS
INTSE Views of
lecture videos
INTSE # Participation
in IVQs
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
C - Quick Evaluation P2P Views of lecture
videos
P2P # Participation in
Quick Evaluation
Results – In-Video-Surveys vs. Quick Evaluation
Inside the video is more effective than outside the video.
MOOCs – A = engineering, B = education, C = physics, D = medicine
Our learners are…
A B C D
Ave. Age (SD)
34.37
(11.37)
39.91
(12.39)
35.9
(15)
37.35
(13.84)
Gender ratio
(M/F)
4/1 1/1.2 2.6/1 1/1.2
Full-time
Employed
57% 63% 59% 59%
Full-time
Studying
41% 30% 39% 38%
Bachelors and
Above
74% 82% 63% 64%
Prior Experience 41.51% 57.20% - 79.25%
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – motivations I
A C D
M(SD) M(SD) M(SD)
For personal growth and enrichment 3.23 (.76) 2.10(.72)
General interest in topic 3.14 (.74) 2.23(.72) 3.23(.92)
Relevant to job 2.63 (1.01) 1.6(.76) 2.71(1.38)
For fun and challenge 2.58 (.96) 1.42(.63)
Earn a certificate/statement of
accomplishment
2.42 (1.03) 1.55 (.70)
Course offered by prestigious
university/professor
2.17 (1.03) 1.91(.77) 3.06(.97)
For career change 2.07 (1.06) 1.32(.59) 1.92(1.41)
Experience an online course 1.81 (.93) 1.21(.51)
Relevant to school or degree program 1.75 (1.00) 1.74(.83) 2.56(1.37)
To improve my English skills 1.66 (.95) 1.52(.71)
Relevant to academic research 1.65 (.98) 2.37(.86)
Meet new people 1.52 (.76) 1.65(.70) 1.96(1.06)
Take with colleagues/friends 1.32 (.68) 1.93(.75)
1=Not at all important, 2 = somewhat important, 3 = very important, 4 = extremely important
Results – Motivations II
Reasons
Item loading
A1
C2
D3
I II III VI I II III VI I II
Experience an online course1,2 0.7 0.7
Meet new people1,2,3 0.7 0.8 0.6
To improve my English skills1,2 0.7 0.2 0.6
Take with colleagues/friends1,2 0.7 0.6
Course offered by prestigious
University/Professor1,2,3
0.6 0.8 0.8
For fun and challenge1,2 0.8 0.7
Personal growth&enrichment1,2 0.8 0.8
General interest in topic1,2,3 0.7 0.8 0.5
Relevant to academic research1,2 0.9 0.8
Relevant to academic study1,2,3 0.9 0.8 0.8
Relevant to job1,2,3 0.8 0.7 0.8
Earn a certificate1,2 0.6
For career change1,2,3 0.5 0.8 0.7
Items with loading lower than 0.5 are removed. Superscripts 1, 2, 3 represent occurrence in A, C, & D respective.
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Course B Specifics
Intent & Goals
Results – Course B Specifics (Cont’d)
2.20%
2.77%
3.72%
3.77%
8.76%
14.17%
30.43%
34.17%
I want to develop my knowledge and
understanding of a specific concept
I want to exchange ideas and learn from
colleagues
I am interested in looking at the course design
I am curious about the topic
I want to complete the course and obtain
recognition and certification for my work
I want to use my knowledge to develop a personal
online teaching strategy
I want to develop my knowledge and
understanding of the overall topic
I want to develop an online learning design that I
can use in my own teaching
(N = 4083)
• Overview of Learners’ Intents
Results – Course B Specifics (Cont’d)
• Intent is not sig. correlated to learners’ behaviours
• Data Mining (K-NN, Naïve Bayes) using learners’ characteristics (incl.
behaviours) indicate 30% predictive accuracy for performance class (no
grade, pass, distinction).
Results – Course B Specifics (cont’d)
Activity by intent
Results – Course B Specifics (Cont’d)
Week 2
Week 4
Week 7
Assessment scores vs. Intents
0
10000
20000
30000
40000
50000
60000
Announcement
Times
Site Access
Video Views &
Downloads
Announcements superimposed on User Video and Site Access Activity
Results – Course B Specifics (Cont’d)
Results – The same in Course D
Results – Engagement with Forum
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Engagement with Quizzes
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Engagement with Videos
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Engagement III
Comparing paying & non-
paying learners
F t df
Sig.
2-tails
Mean
Diff’
CI=95% Interval
of diff'
Lower Upper
Q1 I find this lecture useful.
33.66 -4.9 9092 <0.05 -0.11 -0.16 -0.07
Q2 I understand the content
of this lecture. 24.81 -3.89 9045 <0.05 -0.09 -0.13 -0.04
Q3 I’d like to explore other
modules of this course.
43.62 -5.16 8896 <0.05 -0.11 -0.15 -0.07
Results of F-test suggest unequal variance hence the t-test results assumed unequal variance are used.
(Using Course A as an example – in-video survey items)
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – Learners’ Course Experience
Course experience items (mapped on new CATEI)
A B C
I II I II I II
Overall, I am satisfied with the quality of this course. 0.9 0.7 0.6
The material of the course was presented in an engaging manner. 0.9 0.8 0.8
Overall, this course met my expectations. 0.9 0.8 0.6
Examples, illustrations or real-world cases were used effectively to
explain things.
0.9 0.8 0.7
The course encouraged my interest in this field of study. 0.8 0.8
The goals and requirements of the course were made clear to me. 0.8 0.8 0.7
Overall I have improved the knowledge/skills I will need. 0.8 0.8 0.8
Quizzes helped me to evaluate my progress effectively. 0.6 0.6 0.8
Interacting in the forums helped me to clarify things I did not
understand.
0.9 0.9 0.6
Items with loading lower than 0.5 are removed.
MOOCs – A = engineering, B = education, C = physics, D = medicine
Course experience Items
D
I II
Depth of content 0.8
Course materials 0.8
Lecture videos 0.8
Clarity of explanations 0.8
Course content 0.8
Knowledgeability 0.8
Instructor/teaching staff 0.8
Overall satisfaction 0.7
In-video quizzes 0.7
Presentation skills 0.5
Assessments 0.9
Discussion forums 0.6
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.
a. Rotation converged in 3 iterations.
Results – Learners’ Course Experience (Cont’d)
MOOCs – A = engineering, B = education, C = physics, D = medicine
Results – the Worth of Course Experience
R2 F Sig.
A 0.17 1.22 0.26
C 0.79 0.72 0.72
D 0.39 1.08 0.38
• Motivation cannot predict satisfaction
LinearRegression(motivations) -> ‘How satisfied am I’
MOOCs – A = engineering, B = education, C = physics, D = medicine
Aspects
(Question Block) A coefficient Sig'
Course
Experience
Overall, this course met my expectations 0.54 0.00
The material of the course was presented in an engaging manner 0.19 0.00
The course encouraged my interest in its field of study 0.11 0.00
Examples, illustrations or real-world cases were used effectively to explain
things
0.08 0.01
The goals and requirements of the course were made clear to me 0.07 0.02
Aspects
(Question Block B coefficient Sig'
Course
Experience
Overall this course met my expectations 0.57 0.00
Satisfaction Provided an opportunity for fun and challenge 0.10 0.03
What can predict satisfaction
‘How satisfied am I’ <= LinearRegression(multiple items)
Results – the Worth of Course Experience (Cont’d)
Results – the Worth of Course Experience (Cont’d)
Aspects
(Question Block)
C coefficient Sig'
Overall I have improved the knowledge/skills I will need 0.63 0.00
Course Experience The course encouraged my interest in this field of study 0.61 0.01
The material of the course was presented in an engaging manner 0.32 0.02
The course encouraged my interest in this field of study 0.36 0.02
The experimental tasks were useful to apply the theory 0.30 0.02
LinearRegression(multiple items) ->‘How satisfied am I’
Aspects
(Question Block)
D coefficient Sig'
Self-regulation
I am confident that I could deal efficiently with unexpected events. 0.87 0.00
I can remain calm when facing difficulties because I can rely on my
coping abilities. 0.62 0.01
I can solve most problems if I invest the necessary effort. 0.35 0.02
I can usually handle whatever comes my way 0.32 0.05
Course Experience
Discussion forums 0.42 0.01
In-video quizzes 0.45 0.03
Depth of content 0.67 0.03
I found the material interesting. 0.31 0.03
Motivation
I am taking the course out of general interest, curiosity, or
enjoyment.
0.28 0.03
Conclusions
1. HBR paper*: two types of learners – ‘career
builders’ and ‘education seekers’.
2. Motivation weakly related to engagement or
satisfaction
3. Some relations between behaviors (engagement)
and performance
4. ‘Embedded’ evaluation tools are more effective
5. Mapping of ‘online’ learning experience
* Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting
from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-benefiting-from-
moocs-and-why
Challenges
What are the characteristics of MOOCs that scaffolds
learning?
The intricacy of the combination of the factors which
contribute to the worth of course experience.
The matter of Agency.
Questions and feedbacks
catherine.zhao@unsw.edu.au

More Related Content

What's hot

Platform Showcase: Open2Study. Moodlemoot AU 2013
Platform Showcase: Open2Study. Moodlemoot AU 2013Platform Showcase: Open2Study. Moodlemoot AU 2013
Platform Showcase: Open2Study. Moodlemoot AU 2013
s_dua
 
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
Charles Darwin University
 
Rutgers University CHAMP Evaluation presentation 8 24.14
Rutgers University CHAMP Evaluation presentation 8 24.14Rutgers University CHAMP Evaluation presentation 8 24.14
Rutgers University CHAMP Evaluation presentation 8 24.14
cccschamp
 
Detf update for regional meetings
Detf update for regional meetingsDetf update for regional meetings
Detf update for regional meetings
cccscoetc
 
MCCVLC webinar - Online Program Review
MCCVLC webinar - Online Program ReviewMCCVLC webinar - Online Program Review
MCCVLC webinar - Online Program Review
Barry Dahl
 
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
cccscoetc
 
Keynote Annelies Raes
Keynote Annelies RaesKeynote Annelies Raes
Keynote Annelies Raes
EADTU
 
JISC VRE/Research Tools presentation
JISC VRE/Research Tools presentationJISC VRE/Research Tools presentation
JISC VRE/Research Tools presentation
Christopher Brown
 
Academic Staff Induction (Sept 2015)
Academic Staff Induction (Sept 2015)Academic Staff Induction (Sept 2015)
Academic Staff Induction (Sept 2015)
AQD Winchester
 
A Conversation About the Challenges Facing eLearning Leaders A Review of ITC...
A Conversation About the Challenges Facing eLearning Leaders  A Review of ITC...A Conversation About the Challenges Facing eLearning Leaders  A Review of ITC...
A Conversation About the Challenges Facing eLearning Leaders A Review of ITC...
SmarterServices Owen
 
Overview Jan 2012
Overview Jan 2012Overview Jan 2012
Overview Jan 2012
cccscoetc
 
Keynote Josie Fraser
Keynote Josie FraserKeynote Josie Fraser
Keynote Josie Fraser
EADTU
 
Klaus Dieter Rossade online assessment
Klaus Dieter Rossade online assessmentKlaus Dieter Rossade online assessment
Klaus Dieter Rossade online assessment
EADTU
 
Continuous Improvement in Teaching and Learning – The Community College Open ...
Continuous Improvement in Teaching and Learning – The Community College Open ...Continuous Improvement in Teaching and Learning – The Community College Open ...
Continuous Improvement in Teaching and Learning – The Community College Open ...
Wa State Board Community and Technical Colleges
 
Learners' experiences of innovative 'flipped' and open curricula
Learners' experiences of innovative 'flipped' and open curriculaLearners' experiences of innovative 'flipped' and open curricula
Learners' experiences of innovative 'flipped' and open curricula
ELESIGpresentations
 
Acec 1 15 13
Acec 1 15 13Acec 1 15 13
Acec 1 15 13
cccscoetc
 
Rmcta conference 09 14 12
Rmcta conference 09 14 12Rmcta conference 09 14 12
Rmcta conference 09 14 12
cccscoetc
 
Is Our Program Working? How to Partner with Evaluators and Get Results
Is Our Program Working? How to Partner with Evaluators and Get ResultsIs Our Program Working? How to Partner with Evaluators and Get Results
Is Our Program Working? How to Partner with Evaluators and Get Results
National Partnership for Educational Access
 
DETF recommendations CMC 2013
DETF recommendations CMC 2013DETF recommendations CMC 2013
DETF recommendations CMC 2013
cccscoetc
 
Detf updated jan 2013 for dean group
Detf updated jan 2013 for dean groupDetf updated jan 2013 for dean group
Detf updated jan 2013 for dean group
cccscoetc
 

What's hot (20)

Platform Showcase: Open2Study. Moodlemoot AU 2013
Platform Showcase: Open2Study. Moodlemoot AU 2013Platform Showcase: Open2Study. Moodlemoot AU 2013
Platform Showcase: Open2Study. Moodlemoot AU 2013
 
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
Preparing for an inter-institutional Benchmarking activity using the ACODE Be...
 
Rutgers University CHAMP Evaluation presentation 8 24.14
Rutgers University CHAMP Evaluation presentation 8 24.14Rutgers University CHAMP Evaluation presentation 8 24.14
Rutgers University CHAMP Evaluation presentation 8 24.14
 
Detf update for regional meetings
Detf update for regional meetingsDetf update for regional meetings
Detf update for regional meetings
 
MCCVLC webinar - Online Program Review
MCCVLC webinar - Online Program ReviewMCCVLC webinar - Online Program Review
MCCVLC webinar - Online Program Review
 
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
 
Keynote Annelies Raes
Keynote Annelies RaesKeynote Annelies Raes
Keynote Annelies Raes
 
JISC VRE/Research Tools presentation
JISC VRE/Research Tools presentationJISC VRE/Research Tools presentation
JISC VRE/Research Tools presentation
 
Academic Staff Induction (Sept 2015)
Academic Staff Induction (Sept 2015)Academic Staff Induction (Sept 2015)
Academic Staff Induction (Sept 2015)
 
A Conversation About the Challenges Facing eLearning Leaders A Review of ITC...
A Conversation About the Challenges Facing eLearning Leaders  A Review of ITC...A Conversation About the Challenges Facing eLearning Leaders  A Review of ITC...
A Conversation About the Challenges Facing eLearning Leaders A Review of ITC...
 
Overview Jan 2012
Overview Jan 2012Overview Jan 2012
Overview Jan 2012
 
Keynote Josie Fraser
Keynote Josie FraserKeynote Josie Fraser
Keynote Josie Fraser
 
Klaus Dieter Rossade online assessment
Klaus Dieter Rossade online assessmentKlaus Dieter Rossade online assessment
Klaus Dieter Rossade online assessment
 
Continuous Improvement in Teaching and Learning – The Community College Open ...
Continuous Improvement in Teaching and Learning – The Community College Open ...Continuous Improvement in Teaching and Learning – The Community College Open ...
Continuous Improvement in Teaching and Learning – The Community College Open ...
 
Learners' experiences of innovative 'flipped' and open curricula
Learners' experiences of innovative 'flipped' and open curriculaLearners' experiences of innovative 'flipped' and open curricula
Learners' experiences of innovative 'flipped' and open curricula
 
Acec 1 15 13
Acec 1 15 13Acec 1 15 13
Acec 1 15 13
 
Rmcta conference 09 14 12
Rmcta conference 09 14 12Rmcta conference 09 14 12
Rmcta conference 09 14 12
 
Is Our Program Working? How to Partner with Evaluators and Get Results
Is Our Program Working? How to Partner with Evaluators and Get ResultsIs Our Program Working? How to Partner with Evaluators and Get Results
Is Our Program Working? How to Partner with Evaluators and Get Results
 
DETF recommendations CMC 2013
DETF recommendations CMC 2013DETF recommendations CMC 2013
DETF recommendations CMC 2013
 
Detf updated jan 2013 for dean group
Detf updated jan 2013 for dean groupDetf updated jan 2013 for dean group
Detf updated jan 2013 for dean group
 

Viewers also liked

Presentation slides v2
Presentation slides v2Presentation slides v2
Presentation slides v2
Catherine Zhao
 
Cz la eds-jun-21_v3
Cz la eds-jun-21_v3Cz la eds-jun-21_v3
Cz la eds-jun-21_v3
Catherine Zhao
 
Telefónica movistar
Telefónica movistarTelefónica movistar
Telefónica movistar
Agata Paredes
 
Effective student learning through goal setting v2
Effective student learning through goal setting v2Effective student learning through goal setting v2
Effective student learning through goal setting v2
Catherine Zhao
 
Stephen Geri: Celebrating Autumn in San Antonio
Stephen Geri: Celebrating Autumn in San AntonioStephen Geri: Celebrating Autumn in San Antonio
Stephen Geri: Celebrating Autumn in San Antonio
Stephen Geri
 
DUO-Eduard García-Cruz
DUO-Eduard García-CruzDUO-Eduard García-Cruz
DUO-Eduard García-Cruz
Eduard Garcia Cruz
 
Traditional representation of women
Traditional representation of womenTraditional representation of women
Traditional representation of women
rosslikesjafacakes
 
International Solar Power Expansion by Procorre
International Solar Power Expansion by ProcorreInternational Solar Power Expansion by Procorre
International Solar Power Expansion by Procorre
Procorre
 
How Do You Compare to Serious Business Bloggers?
How Do You Compare to Serious Business Bloggers?How Do You Compare to Serious Business Bloggers?
How Do You Compare to Serious Business Bloggers?
Barry Feldman
 
The Power of the Hashtag - Social Media as an Educational Imperative
The Power of the Hashtag - Social Media as an Educational ImperativeThe Power of the Hashtag - Social Media as an Educational Imperative
The Power of the Hashtag - Social Media as an Educational Imperative
Christopher Casal
 
Bart Beasley Custom Builders: 2809 Middle St.
Bart Beasley Custom Builders: 2809 Middle St.Bart Beasley Custom Builders: 2809 Middle St.
Bart Beasley Custom Builders: 2809 Middle St.
Bart Beasley
 
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactosPEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
Marcelo Bancalero
 
How to Improve Company Culture
How to Improve Company CultureHow to Improve Company Culture
How to Improve Company Culture
Matt Kupec
 
Vipin kumar gour
Vipin kumar gourVipin kumar gour
Vipin kumar gour
Vipin Kumar Gour
 
No estimates - a controversial way to improve estimation with results-handouts
No estimates - a controversial way to improve estimation with results-handoutsNo estimates - a controversial way to improve estimation with results-handouts
No estimates - a controversial way to improve estimation with results-handouts
Vasco Duarte
 
Weidenhammer Digital Transformation Presentation
Weidenhammer Digital Transformation PresentationWeidenhammer Digital Transformation Presentation
Weidenhammer Digital Transformation Presentation
Gene Ferro
 
Considerations and examples for key performance indicators design
Considerations and examples for key performance indicators designConsiderations and examples for key performance indicators design
Considerations and examples for key performance indicators design
JAMSO
 

Viewers also liked (17)

Presentation slides v2
Presentation slides v2Presentation slides v2
Presentation slides v2
 
Cz la eds-jun-21_v3
Cz la eds-jun-21_v3Cz la eds-jun-21_v3
Cz la eds-jun-21_v3
 
Telefónica movistar
Telefónica movistarTelefónica movistar
Telefónica movistar
 
Effective student learning through goal setting v2
Effective student learning through goal setting v2Effective student learning through goal setting v2
Effective student learning through goal setting v2
 
Stephen Geri: Celebrating Autumn in San Antonio
Stephen Geri: Celebrating Autumn in San AntonioStephen Geri: Celebrating Autumn in San Antonio
Stephen Geri: Celebrating Autumn in San Antonio
 
DUO-Eduard García-Cruz
DUO-Eduard García-CruzDUO-Eduard García-Cruz
DUO-Eduard García-Cruz
 
Traditional representation of women
Traditional representation of womenTraditional representation of women
Traditional representation of women
 
International Solar Power Expansion by Procorre
International Solar Power Expansion by ProcorreInternational Solar Power Expansion by Procorre
International Solar Power Expansion by Procorre
 
How Do You Compare to Serious Business Bloggers?
How Do You Compare to Serious Business Bloggers?How Do You Compare to Serious Business Bloggers?
How Do You Compare to Serious Business Bloggers?
 
The Power of the Hashtag - Social Media as an Educational Imperative
The Power of the Hashtag - Social Media as an Educational ImperativeThe Power of the Hashtag - Social Media as an Educational Imperative
The Power of the Hashtag - Social Media as an Educational Imperative
 
Bart Beasley Custom Builders: 2809 Middle St.
Bart Beasley Custom Builders: 2809 Middle St.Bart Beasley Custom Builders: 2809 Middle St.
Bart Beasley Custom Builders: 2809 Middle St.
 
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactosPEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
PEC nº 241/2016: o novo regime fiscal e seus possíveis impactos
 
How to Improve Company Culture
How to Improve Company CultureHow to Improve Company Culture
How to Improve Company Culture
 
Vipin kumar gour
Vipin kumar gourVipin kumar gour
Vipin kumar gour
 
No estimates - a controversial way to improve estimation with results-handouts
No estimates - a controversial way to improve estimation with results-handoutsNo estimates - a controversial way to improve estimation with results-handouts
No estimates - a controversial way to improve estimation with results-handouts
 
Weidenhammer Digital Transformation Presentation
Weidenhammer Digital Transformation PresentationWeidenhammer Digital Transformation Presentation
Weidenhammer Digital Transformation Presentation
 
Considerations and examples for key performance indicators design
Considerations and examples for key performance indicators designConsiderations and examples for key performance indicators design
Considerations and examples for key performance indicators design
 

Similar to Wsu guest lecture_2015_czlv_v3

STEM tech presentation TriC MATH MOOC
STEM tech presentation TriC MATH MOOCSTEM tech presentation TriC MATH MOOC
STEM tech presentation TriC MATH MOOC
Cheryl Thomas Knight
 
The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator
Simon Bates
 
Student dropout in distance education - how many, who, when, what are the co...
Student dropout in distance education  - how many, who, when, what are the co...Student dropout in distance education  - how many, who, when, what are the co...
Student dropout in distance education - how many, who, when, what are the co...
EADTU
 
Parallel Trajectories: Distance Learning, OER, and Scaling Innovation
Parallel Trajectories: Distance Learning, OER, and Scaling InnovationParallel Trajectories: Distance Learning, OER, and Scaling Innovation
Parallel Trajectories: Distance Learning, OER, and Scaling Innovation
Achieving the Dream
 
Royal Society for Biology: Using tests and badges
Royal Society for Biology: Using tests and badgesRoyal Society for Biology: Using tests and badges
Royal Society for Biology: Using tests and badges
University of Derby
 
2013 QMOD Presentation
2013 QMOD Presentation 2013 QMOD Presentation
2013 QMOD Presentation
Brandon Theiss, PE
 
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
Blackboard APAC
 
Garbage in
Garbage inGarbage in
Garbage in
bbridges51
 
An overview of SLN\'s award-winning Faculty Development Process: 5 key elements
An overview of SLN\'s award-winning Faculty Development Process: 5 key elementsAn overview of SLN\'s award-winning Faculty Development Process: 5 key elements
An overview of SLN\'s award-winning Faculty Development Process: 5 key elements
Alexandra M. Pickett
 
Learning analytics on MOOCs
Learning analytics on MOOCsLearning analytics on MOOCs
Learning analytics on MOOCs
Fabrice Demichel
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
EDEN Digital Learning Europe
 
Ex science 11nov
Ex science 11novEx science 11nov
Ex science 11nov
Susie Macfarlane
 
Quality, Who Says
Quality, Who SaysQuality, Who Says
Quality, Who Says
WCET Conference 2008
 
Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning Programming
Peter Brusilovsky
 
Educational Question Routing in Online Student Communities
 Educational Question Routing in Online Student Communities Educational Question Routing in Online Student Communities
Educational Question Routing in Online Student Communities
Jakub Macina
 
Pedagogy with Technology: getting the horse out in front of the cart
Pedagogy with Technology: getting the horse out in front of the cartPedagogy with Technology: getting the horse out in front of the cart
Pedagogy with Technology: getting the horse out in front of the cart
Simon Bates
 
Implementing Open Badges in Three Preservice Teacher Education Programs 2015
Implementing Open Badges in Three Preservice Teacher Education Programs 2015Implementing Open Badges in Three Preservice Teacher Education Programs 2015
Implementing Open Badges in Three Preservice Teacher Education Programs 2015
Dan Randall
 
Program outcome maping
Program outcome mapingProgram outcome maping
Program outcome maping
Vijay Kumar Jadon
 
Key elements in successful online faculty development
Key elements in successful online faculty developmentKey elements in successful online faculty development
Key elements in successful online faculty development
Alexandra M. Pickett
 
How Do You Get There From Here
How Do You Get There From HereHow Do You Get There From Here
How Do You Get There From Here
WCET
 

Similar to Wsu guest lecture_2015_czlv_v3 (20)

STEM tech presentation TriC MATH MOOC
STEM tech presentation TriC MATH MOOCSTEM tech presentation TriC MATH MOOC
STEM tech presentation TriC MATH MOOC
 
The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator
 
Student dropout in distance education - how many, who, when, what are the co...
Student dropout in distance education  - how many, who, when, what are the co...Student dropout in distance education  - how many, who, when, what are the co...
Student dropout in distance education - how many, who, when, what are the co...
 
Parallel Trajectories: Distance Learning, OER, and Scaling Innovation
Parallel Trajectories: Distance Learning, OER, and Scaling InnovationParallel Trajectories: Distance Learning, OER, and Scaling Innovation
Parallel Trajectories: Distance Learning, OER, and Scaling Innovation
 
Royal Society for Biology: Using tests and badges
Royal Society for Biology: Using tests and badgesRoyal Society for Biology: Using tests and badges
Royal Society for Biology: Using tests and badges
 
2013 QMOD Presentation
2013 QMOD Presentation 2013 QMOD Presentation
2013 QMOD Presentation
 
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
Online Tests: Can we do them better? | Bopelo Boitshwarelo, Jyoti Vemuri, Han...
 
Garbage in
Garbage inGarbage in
Garbage in
 
An overview of SLN\'s award-winning Faculty Development Process: 5 key elements
An overview of SLN\'s award-winning Faculty Development Process: 5 key elementsAn overview of SLN\'s award-winning Faculty Development Process: 5 key elements
An overview of SLN\'s award-winning Faculty Development Process: 5 key elements
 
Learning analytics on MOOCs
Learning analytics on MOOCsLearning analytics on MOOCs
Learning analytics on MOOCs
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
 
Ex science 11nov
Ex science 11novEx science 11nov
Ex science 11nov
 
Quality, Who Says
Quality, Who SaysQuality, Who Says
Quality, Who Says
 
Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning Programming
 
Educational Question Routing in Online Student Communities
 Educational Question Routing in Online Student Communities Educational Question Routing in Online Student Communities
Educational Question Routing in Online Student Communities
 
Pedagogy with Technology: getting the horse out in front of the cart
Pedagogy with Technology: getting the horse out in front of the cartPedagogy with Technology: getting the horse out in front of the cart
Pedagogy with Technology: getting the horse out in front of the cart
 
Implementing Open Badges in Three Preservice Teacher Education Programs 2015
Implementing Open Badges in Three Preservice Teacher Education Programs 2015Implementing Open Badges in Three Preservice Teacher Education Programs 2015
Implementing Open Badges in Three Preservice Teacher Education Programs 2015
 
Program outcome maping
Program outcome mapingProgram outcome maping
Program outcome maping
 
Key elements in successful online faculty development
Key elements in successful online faculty developmentKey elements in successful online faculty development
Key elements in successful online faculty development
 
How Do You Get There From Here
How Do You Get There From HereHow Do You Get There From Here
How Do You Get There From Here
 

Recently uploaded

How to Fix [Errno 98] address already in use
How to Fix [Errno 98] address already in useHow to Fix [Errno 98] address already in use
How to Fix [Errno 98] address already in use
Celine George
 
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
220711130100 udita Chakraborty  Aims and objectives of national policy on inf...220711130100 udita Chakraborty  Aims and objectives of national policy on inf...
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
Kalna College
 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Henry Hollis
 
Haunted Houses by H W Longfellow for class 10
Haunted Houses by H W Longfellow for class 10Haunted Houses by H W Longfellow for class 10
Haunted Houses by H W Longfellow for class 10
nitinpv4ai
 
skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)
Mohammad Al-Dhahabi
 
220711130083 SUBHASHREE RAKSHIT Internet resources for social science
220711130083 SUBHASHREE RAKSHIT  Internet resources for social science220711130083 SUBHASHREE RAKSHIT  Internet resources for social science
220711130083 SUBHASHREE RAKSHIT Internet resources for social science
Kalna College
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
سمير بسيوني
 
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
andagarcia212
 
adjectives.ppt for class 1 to 6, grammar
adjectives.ppt for class 1 to 6, grammaradjectives.ppt for class 1 to 6, grammar
adjectives.ppt for class 1 to 6, grammar
7DFarhanaMohammed
 
220711130088 Sumi Basak Virtual University EPC 3.pptx
220711130088 Sumi Basak Virtual University EPC 3.pptx220711130088 Sumi Basak Virtual University EPC 3.pptx
220711130088 Sumi Basak Virtual University EPC 3.pptx
Kalna College
 
SWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptxSWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptx
zuzanka
 
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
indexPub
 
Contiguity Of Various Message Forms - Rupam Chandra.pptx
Contiguity Of Various Message Forms - Rupam Chandra.pptxContiguity Of Various Message Forms - Rupam Chandra.pptx
Contiguity Of Various Message Forms - Rupam Chandra.pptx
Kalna College
 
FinalSD_MathematicsGrade7_Session2_Unida.pptx
FinalSD_MathematicsGrade7_Session2_Unida.pptxFinalSD_MathematicsGrade7_Session2_Unida.pptx
FinalSD_MathematicsGrade7_Session2_Unida.pptx
JennySularte1
 
Creation or Update of a Mandatory Field is Not Set in Odoo 17
Creation or Update of a Mandatory Field is Not Set in Odoo 17Creation or Update of a Mandatory Field is Not Set in Odoo 17
Creation or Update of a Mandatory Field is Not Set in Odoo 17
Celine George
 
Observational Learning
Observational Learning Observational Learning
Observational Learning
sanamushtaq922
 
Ch-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdfCh-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdf
lakshayrojroj
 
Educational Technology in the Health Sciences
Educational Technology in the Health SciencesEducational Technology in the Health Sciences
Educational Technology in the Health Sciences
Iris Thiele Isip-Tan
 
Creative Restart 2024: Mike Martin - Finding a way around “no”
Creative Restart 2024: Mike Martin - Finding a way around “no”Creative Restart 2024: Mike Martin - Finding a way around “no”
Creative Restart 2024: Mike Martin - Finding a way around “no”
Taste
 
How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17
Celine George
 

Recently uploaded (20)

How to Fix [Errno 98] address already in use
How to Fix [Errno 98] address already in useHow to Fix [Errno 98] address already in use
How to Fix [Errno 98] address already in use
 
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
220711130100 udita Chakraborty  Aims and objectives of national policy on inf...220711130100 udita Chakraborty  Aims and objectives of national policy on inf...
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
 
Haunted Houses by H W Longfellow for class 10
Haunted Houses by H W Longfellow for class 10Haunted Houses by H W Longfellow for class 10
Haunted Houses by H W Longfellow for class 10
 
skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)
 
220711130083 SUBHASHREE RAKSHIT Internet resources for social science
220711130083 SUBHASHREE RAKSHIT  Internet resources for social science220711130083 SUBHASHREE RAKSHIT  Internet resources for social science
220711130083 SUBHASHREE RAKSHIT Internet resources for social science
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
 
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
欧洲杯下注-欧洲杯下注押注官网-欧洲杯下注押注网站|【​网址​🎉ac44.net🎉​】
 
adjectives.ppt for class 1 to 6, grammar
adjectives.ppt for class 1 to 6, grammaradjectives.ppt for class 1 to 6, grammar
adjectives.ppt for class 1 to 6, grammar
 
220711130088 Sumi Basak Virtual University EPC 3.pptx
220711130088 Sumi Basak Virtual University EPC 3.pptx220711130088 Sumi Basak Virtual University EPC 3.pptx
220711130088 Sumi Basak Virtual University EPC 3.pptx
 
SWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptxSWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptx
 
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...
 
Contiguity Of Various Message Forms - Rupam Chandra.pptx
Contiguity Of Various Message Forms - Rupam Chandra.pptxContiguity Of Various Message Forms - Rupam Chandra.pptx
Contiguity Of Various Message Forms - Rupam Chandra.pptx
 
FinalSD_MathematicsGrade7_Session2_Unida.pptx
FinalSD_MathematicsGrade7_Session2_Unida.pptxFinalSD_MathematicsGrade7_Session2_Unida.pptx
FinalSD_MathematicsGrade7_Session2_Unida.pptx
 
Creation or Update of a Mandatory Field is Not Set in Odoo 17
Creation or Update of a Mandatory Field is Not Set in Odoo 17Creation or Update of a Mandatory Field is Not Set in Odoo 17
Creation or Update of a Mandatory Field is Not Set in Odoo 17
 
Observational Learning
Observational Learning Observational Learning
Observational Learning
 
Ch-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdfCh-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdf
 
Educational Technology in the Health Sciences
Educational Technology in the Health SciencesEducational Technology in the Health Sciences
Educational Technology in the Health Sciences
 
Creative Restart 2024: Mike Martin - Finding a way around “no”
Creative Restart 2024: Mike Martin - Finding a way around “no”Creative Restart 2024: Mike Martin - Finding a way around “no”
Creative Restart 2024: Mike Martin - Finding a way around “no”
 
How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17
 

Wsu guest lecture_2015_czlv_v3

  • 1. Evaluating the ‘Student’ Experience in Massive Open Online Courses (MOOCs) Catherine Zhao & Lorenzo Vigentini Learning and Teaching Unit, UNSW Australia
  • 2. In this presentation - • Background – Previous work on Student learning Experience & UNSW MOOCs • Methods - the surveys for evaluation • Findings - o Surveys - fit-for-purpose? o Learners – what’s it like to experience a MOOC? • Current work & Challenges
  • 3. A Model of the Student Learning Experience Worth of Experience Future Past Motivation Experience Perception Engagement Aspirations Expectations Personal development ‘match’ or ‘fit’ with context/LE Direct future benefits Vigentini, L., and Zhao, C. Revisiting end-of-semester evaluation: the questions, the instrument and the process.,” in AAIR : turning silver into gold, Melbourne, 2014.
  • 4. Background – Why UNSW joined the ‘bandwagon’
  • 5. Background- What it takes to design and deliver a MOOC
  • 6. Five known facts about MOOCs Massive enrollments Dramatic dropouts Unconventional learners (Relatively) limited design/delivery options Engagement and the learner experience The rise of ‘certificates’
  • 7.
  • 8. Background - Four MOOCs we look at A B C D Disciplines Engineering Education Science Medicine Duration (weeks) 9 6 9 8 Target Group Engineers Teachers High school students/ teachers General public Course Design All-at-once All-at-once Sequential Semi-sequential Course Delivery All-at-once Staggered Staggered Staggered Videos 110 224 98 58 Quizzes 10 22 42 2 Assignments 7 3 2 2 Forums 54 105 63 64 Registrants (Active learners*) 32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%) *Active learner: those who appear at least once during the course MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 9. Background- What & When Surveys Evaluation Surveys Deployment A B C D Demographic survey Prior to course commencement     Pre-course survey OR First course activity     In-video surveys (IVS) During the course  Quick evaluation (Video)  Quick evaluation (Module)  Quick Question survey   Post-course survey End of course      Present in MOOC MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 10. Methods - Overview of the Pre-Course survey questions Common A B C D Course-specific A B C D Reasons/Purpose of Joining    Background-Gender, Ethnicity  Prior Professional Experience/Knowledge     English Proficiency  Intended Effort    Prior MOOC Experience  Familiarisation with Topics   Geographic Location  Academic Qualification    Contributing Elements of ‘Teaching Online’  Industry of Employment   Defining ‘Completion'  Preventing Factors for Course Completion   Learning Preference  Expected Outcomes (course related)   Achievement Goal Oriented   Present in the course MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 11. Common A B C D Course-specific A B C D Course Satisfaction     Proportion of Completed Course Activities  Course Features     Evaluating Peer Assessment  Actual Effort     (Probing) Self-Regulated Learning  Desired Topics    Satisfaction on Personal Development    (Intend to) Apply Course Topics     Considering Credit-Bearing Courses   Methods- Overview of the Post-course survey questions MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 12. Methods– Pre- & Post-Course Survey Sample Sizes A B C D # Active Learners 19708 18024 10576 4673 Required sample size* 2141 2119 1957 1587 Pre- & Post- Course survey Pre-course survey sample 2684 4490 264 401 response rate 13.62% 24.91% 2.50% 8.58% Margin of error 1.76% 1.28% 5.96% 4.68% Post-course survey sample 638 313 126 71 response rate 3.24% 1.74% 1.19% 1.52% Margin of error 3.82% 5.49% 8.68% 11.54% Coursera Demographic Survey # of registrants 32928 28864 22487 13185 Demographic Survey Responses 2774 2873 2720 1466 Margin of error 1.20% 2.20% 1.52% 2.41% *Required sample size is calculated under the condition of CL = 95%, and margin of error = 2%
  • 13. Methods – Patterns of participation in Pre-& Post- Course surveys Counts of responses in the four MOOCs’ pre-course surveys (logarithmic scale) over 9 weeks. 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 Countofresponses A Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 B Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 Countofresponses C Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 D Pre Post
  • 14. Methods- Collection tools Course A IVS (In-video survey) Course C Quick Evaluation MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 15. Research Questions 1) Are the tools used robust enough to characterize the learners’ experience in MOOC? 2) Are there substantial differences in the modes of gathering feedback? 3) Is it possible to identify key elements of the learners’ MOOC experiences? 4) What is the relevance of ‘teaching’ questions for the online experience?
  • 16. Results – Are the surveys robust? • Accuracy • A B C D Required sample size (CI=95%,margin of error 2%) 2141 2119 1957 1587 Pre-course survey sample 2684 4490 264 401 Margin of error 1.76% 1.28% 5.96% 4.68% Post-course survey sample 638 313 126 71 Margin of error 3.82% 5.49% 8.68% 11.54%
  • 17. Results – Are the surveys robust? (Cont’d) A B C D Motivation Block 0.78 0.86 0.65 Course experience Block 0.90 0.94 0.89 0.92 IVS items 0.82 Confidence 0.84 Satisfaction of personal gain 0.91 Satisfaction of course materials and personal gain 0.92 (probing) self-regulated learning skills 0.90 • Reliability – Cronbach’s alpha MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 18. Results – Are the surveys robust? (Cont’d) • For the motivation block that is <.7 (D) Reason items Cronbach's Alpha if Item Deleted I am taking the course out of general interest, curiosity, or enjoyment. .722 I am interested in taking a course from this particular institution. .616 I am interested in learning from the expert researchers and clinicians delivering the course. .636 I want to connect with other students interested in this topic. .589 The course supports my current academic program. .565 The course is aligned with my current job responsibilities or company's line-of-business. .588 The skills gained from this course may be useful for obtaining a new job. .571 MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 19. Results – Comparison of Evaluation Approaches • The reasons for adopting different approaches 1. Technical constraints 2. Accommodating various requirements put forward by academic leads 3. Test different tools/ways to evaluate • Two key sets of approaches we are comparing - 1. First course activity in A & B vs. Pre-course survey in C & D 2. In-Video Surveys (IVS) vs. Quick Evaluation (QE) MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 20. Results – Effectiveness of Pre-course survey • ‘Embedded’ (= first course activity) is more effective than ‘stand-alone’ • Higher response rate • measures how much learners want to engage A B C D 13.62% 24.91% 2.50% 8.58% MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 21. 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 A - IVS INTSE Views of lecture videos INTSE # Participation in IVQs 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 C - Quick Evaluation P2P Views of lecture videos P2P # Participation in Quick Evaluation Results – In-Video-Surveys vs. Quick Evaluation Inside the video is more effective than outside the video. MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 22. Our learners are… A B C D Ave. Age (SD) 34.37 (11.37) 39.91 (12.39) 35.9 (15) 37.35 (13.84) Gender ratio (M/F) 4/1 1/1.2 2.6/1 1/1.2 Full-time Employed 57% 63% 59% 59% Full-time Studying 41% 30% 39% 38% Bachelors and Above 74% 82% 63% 64% Prior Experience 41.51% 57.20% - 79.25% MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 23. Results – motivations I A C D M(SD) M(SD) M(SD) For personal growth and enrichment 3.23 (.76) 2.10(.72) General interest in topic 3.14 (.74) 2.23(.72) 3.23(.92) Relevant to job 2.63 (1.01) 1.6(.76) 2.71(1.38) For fun and challenge 2.58 (.96) 1.42(.63) Earn a certificate/statement of accomplishment 2.42 (1.03) 1.55 (.70) Course offered by prestigious university/professor 2.17 (1.03) 1.91(.77) 3.06(.97) For career change 2.07 (1.06) 1.32(.59) 1.92(1.41) Experience an online course 1.81 (.93) 1.21(.51) Relevant to school or degree program 1.75 (1.00) 1.74(.83) 2.56(1.37) To improve my English skills 1.66 (.95) 1.52(.71) Relevant to academic research 1.65 (.98) 2.37(.86) Meet new people 1.52 (.76) 1.65(.70) 1.96(1.06) Take with colleagues/friends 1.32 (.68) 1.93(.75) 1=Not at all important, 2 = somewhat important, 3 = very important, 4 = extremely important
  • 24. Results – Motivations II Reasons Item loading A1 C2 D3 I II III VI I II III VI I II Experience an online course1,2 0.7 0.7 Meet new people1,2,3 0.7 0.8 0.6 To improve my English skills1,2 0.7 0.2 0.6 Take with colleagues/friends1,2 0.7 0.6 Course offered by prestigious University/Professor1,2,3 0.6 0.8 0.8 For fun and challenge1,2 0.8 0.7 Personal growth&enrichment1,2 0.8 0.8 General interest in topic1,2,3 0.7 0.8 0.5 Relevant to academic research1,2 0.9 0.8 Relevant to academic study1,2,3 0.9 0.8 0.8 Relevant to job1,2,3 0.8 0.7 0.8 Earn a certificate1,2 0.6 For career change1,2,3 0.5 0.8 0.7 Items with loading lower than 0.5 are removed. Superscripts 1, 2, 3 represent occurrence in A, C, & D respective. MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 25. Results – Course B Specifics Intent & Goals
  • 26. Results – Course B Specifics (Cont’d) 2.20% 2.77% 3.72% 3.77% 8.76% 14.17% 30.43% 34.17% I want to develop my knowledge and understanding of a specific concept I want to exchange ideas and learn from colleagues I am interested in looking at the course design I am curious about the topic I want to complete the course and obtain recognition and certification for my work I want to use my knowledge to develop a personal online teaching strategy I want to develop my knowledge and understanding of the overall topic I want to develop an online learning design that I can use in my own teaching (N = 4083) • Overview of Learners’ Intents
  • 27. Results – Course B Specifics (Cont’d) • Intent is not sig. correlated to learners’ behaviours • Data Mining (K-NN, Naïve Bayes) using learners’ characteristics (incl. behaviours) indicate 30% predictive accuracy for performance class (no grade, pass, distinction).
  • 28. Results – Course B Specifics (cont’d) Activity by intent
  • 29. Results – Course B Specifics (Cont’d) Week 2 Week 4 Week 7 Assessment scores vs. Intents
  • 30. 0 10000 20000 30000 40000 50000 60000 Announcement Times Site Access Video Views & Downloads Announcements superimposed on User Video and Site Access Activity Results – Course B Specifics (Cont’d)
  • 31. Results – The same in Course D
  • 32. Results – Engagement with Forum A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 33.
  • 34. Results – Engagement with Quizzes A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 35. Results – Engagement with Videos A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 36. Results – Engagement III Comparing paying & non- paying learners F t df Sig. 2-tails Mean Diff’ CI=95% Interval of diff' Lower Upper Q1 I find this lecture useful. 33.66 -4.9 9092 <0.05 -0.11 -0.16 -0.07 Q2 I understand the content of this lecture. 24.81 -3.89 9045 <0.05 -0.09 -0.13 -0.04 Q3 I’d like to explore other modules of this course. 43.62 -5.16 8896 <0.05 -0.11 -0.15 -0.07 Results of F-test suggest unequal variance hence the t-test results assumed unequal variance are used. (Using Course A as an example – in-video survey items) MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 37. Results – Learners’ Course Experience Course experience items (mapped on new CATEI) A B C I II I II I II Overall, I am satisfied with the quality of this course. 0.9 0.7 0.6 The material of the course was presented in an engaging manner. 0.9 0.8 0.8 Overall, this course met my expectations. 0.9 0.8 0.6 Examples, illustrations or real-world cases were used effectively to explain things. 0.9 0.8 0.7 The course encouraged my interest in this field of study. 0.8 0.8 The goals and requirements of the course were made clear to me. 0.8 0.8 0.7 Overall I have improved the knowledge/skills I will need. 0.8 0.8 0.8 Quizzes helped me to evaluate my progress effectively. 0.6 0.6 0.8 Interacting in the forums helped me to clarify things I did not understand. 0.9 0.9 0.6 Items with loading lower than 0.5 are removed. MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 38. Course experience Items D I II Depth of content 0.8 Course materials 0.8 Lecture videos 0.8 Clarity of explanations 0.8 Course content 0.8 Knowledgeability 0.8 Instructor/teaching staff 0.8 Overall satisfaction 0.7 In-video quizzes 0.7 Presentation skills 0.5 Assessments 0.9 Discussion forums 0.6 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 3 iterations. Results – Learners’ Course Experience (Cont’d) MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 39. Results – the Worth of Course Experience R2 F Sig. A 0.17 1.22 0.26 C 0.79 0.72 0.72 D 0.39 1.08 0.38 • Motivation cannot predict satisfaction LinearRegression(motivations) -> ‘How satisfied am I’ MOOCs – A = engineering, B = education, C = physics, D = medicine
  • 40. Aspects (Question Block) A coefficient Sig' Course Experience Overall, this course met my expectations 0.54 0.00 The material of the course was presented in an engaging manner 0.19 0.00 The course encouraged my interest in its field of study 0.11 0.00 Examples, illustrations or real-world cases were used effectively to explain things 0.08 0.01 The goals and requirements of the course were made clear to me 0.07 0.02 Aspects (Question Block B coefficient Sig' Course Experience Overall this course met my expectations 0.57 0.00 Satisfaction Provided an opportunity for fun and challenge 0.10 0.03 What can predict satisfaction ‘How satisfied am I’ <= LinearRegression(multiple items) Results – the Worth of Course Experience (Cont’d)
  • 41. Results – the Worth of Course Experience (Cont’d) Aspects (Question Block) C coefficient Sig' Overall I have improved the knowledge/skills I will need 0.63 0.00 Course Experience The course encouraged my interest in this field of study 0.61 0.01 The material of the course was presented in an engaging manner 0.32 0.02 The course encouraged my interest in this field of study 0.36 0.02 The experimental tasks were useful to apply the theory 0.30 0.02 LinearRegression(multiple items) ->‘How satisfied am I’ Aspects (Question Block) D coefficient Sig' Self-regulation I am confident that I could deal efficiently with unexpected events. 0.87 0.00 I can remain calm when facing difficulties because I can rely on my coping abilities. 0.62 0.01 I can solve most problems if I invest the necessary effort. 0.35 0.02 I can usually handle whatever comes my way 0.32 0.05 Course Experience Discussion forums 0.42 0.01 In-video quizzes 0.45 0.03 Depth of content 0.67 0.03 I found the material interesting. 0.31 0.03 Motivation I am taking the course out of general interest, curiosity, or enjoyment. 0.28 0.03
  • 42. Conclusions 1. HBR paper*: two types of learners – ‘career builders’ and ‘education seekers’. 2. Motivation weakly related to engagement or satisfaction 3. Some relations between behaviors (engagement) and performance 4. ‘Embedded’ evaluation tools are more effective 5. Mapping of ‘online’ learning experience * Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-benefiting-from- moocs-and-why
  • 43. Challenges What are the characteristics of MOOCs that scaffolds learning? The intricacy of the combination of the factors which contribute to the worth of course experience. The matter of Agency.

Editor's Notes

  1. Student experience is too broad, includes campus life and you may get questions Services are supporting it, but focus is on learning experience Be prepared
  2. Bridging slide for everyone
  3. Give them context Tell them what A, B, C.. Explain multiple iterations 3x intse 2x ltto 2x p2p 1x Pmed HL next in line
  4. CATEI Is eval of teaching for management, so we test some questions
  5. Say its relatively small
  6. Call it how keen they are to do stuff in the mooc instead. NEED to run an ANOVA on N responses Factors/levels: Credit (y/n) x course (4) x timing (1st week/later)
  7. Correlated, NOT related suggesting low/weak association Separate things! Otherwise you have no idea what it is
  8. Question, what’s the best way of presenting/delivering content? Left is bigger pic of activity for the course: patterns of access for the course as a whole: distinct access people added up for class % of activity per row timeline for each module with spikes each week (traffic) another representation including everything for a module, then you break it down BOTTOM is goal setting activity done all at the start Right is splitting by intents activity by intents over time (they are the same)
  9. Need to explain a bit Learners more likely to participate at the beginning regardless of intents Too many dimensions
  10. Example of how ‘teaching’ drives activity Whenever you have an announcement spikes of activity follow (LTTO)
  11. Breaking down by activity type over multiple courses.
  12. Very dangerous No performance difference in free Scrap this, otherwise you need the figure for 4 courses We will edit later… weeks
  13. This is important, highlight mapping to the new CATEI
  14. Where do you say that it is course B? explain
  15. Deceiving Motivation predicts satisfaction not the other way Why highlight NS? What is C? large R not sure about result
  16. Same here These are sig, but no highlight Relatively large
  17. Same swap of vars large
  18. Harvard business review paper Is it supported? Then explain 2 explain more 3. Give more examples