SlideShare a Scribd company logo
1 of 36
Barry Dahl
Teaching & Learning Advocate, D2L
Five Important Things
You Won't Find in a
Course Quality Rubric
Brief History of Online Course Design Rubrics
•2002: Cal State Chico developed the ROI (Rubric for
Online Instruction)
• Implemented at Chico in January 2003
•2003: FIPSE Grant won by Maryland Online: "Quality
Matters: Inter-Institutional Quality Assurance in Online
Learning“
• Grant term: Sept 2003 – Aug 2006
A Brief Personal History
• Feb. 2004 - QM workshop at ITC eLearning Conference
• May 2004 - Faculty at my college voted to create a peer
review process using V1 of QM rubric as the course design
recommendations
• 2005-present – former college still uses a peer review
process with a rubric adapted from V1 of QM.
• 2010 – began consulting gigs promoting the use of a three-
pronged approach to improving online course and program
quality
• 2012 – began work at D2L – and here we are today
3 Major Components of e-Quality
Learning
Teaching Design
Measuring the e-Quality Components
Learning
Teaching Design
Learning Assessment
Quality of
Quality of Quality of
Learning
Teaching Design
Learning Assessment
Quality of
Quality of Quality of
We’ll
leave this
to
everyone
else
The Other Presentations
5 Items for Today
Learning
Teaching Design
Learning Assessment
Item
#2
Item
#1
Item
#3
Item
#4
Item
#5
Quality of Instruction
Learning
Teaching Design
Teaching
Item
#1
1. Online
Instructor
Evaluation
To determine the
“quality” of the
instruction provided, an
evaluation needs to be
conducted.
Item
#1
Purposes of Instructor Evaluation
Encourage
excellence in
teaching and
learning.
01
Facilitate long-
term quality
improvement by
monitoring
instructional
performance.
02
Provide
constructive
feedback by
identifying
strengths and
areas for
improvement.
03
Inspire
professional
growth and
development.
04
Faculty Evaluation Components
• Student Evaluation of Instruction
• Faculty Self-Evaluation
• Professional Development Plan
• Faculty Peer Review
• Faculty Portfolio
• Supervisor’s Evaluation of Faculty
Typically some combination of the above components
Barry’s
Thoughts on
Online
Instructor
Evaluation
• It’s not the same
• Classroom observations in
traditional classroom are
mostly for show
• Observers/Evaluators need
to “get” the art of online
teaching
• It CAN be done.
Item
#1
Quality (Level) of Learning Achieved
Learning
Teaching Design
Item
#2
Learning
2. Online
Student
Collegewide
Outcomes
Assessment
Similar achievement as F2F?
Are online-only students
included in your collegewide
assessments?
Have your accreditors asked
to see this?
Item
#2
Examples of Collegewide Assessments
•Collegewide Outcome #1
• Written and oral
communication
• Practiced extensively,
across the curriculum, in
the context of
progressively more
challenging problems,
projects, and standards
for performance
• Assessment Projects for #1
• Assessments of writing
skills
• Scored by a cross-
disciplinary team of faculty
• Administered to random
selection of students from
across the college
• Segregate writing samples
of online students for
comparison to on-campus
students
2. Course
Outcomes
Assessment
Comparisons
Similar achievement as F2F?
What can you learn from
this?
What to do about a lower
performing group?
Item
#2
Barry’s
Thoughts on
Quality of
Learning
•Culture of Evidence
•Answer the skeptics
•Don’t say “we can’t”
•College-wide
assessments for
accreditors
Item
#2
Why Do Students Take Online
Courses?
Caveat: assuming they aren’t forced to by a
worldwide pandemic or something similar…
Ruffalo Noel-Levitz PSOL
•This survey can tell you how satisfiedyour students are and what issues are
reallyimportantto them in the areas of:
•Academic services
•Enrollmentservices
•Institutionalperceptions
•Instructionalservices
•Studentservices
PSOL Factors to Enroll in Online
46. Financialassistanceavailable
47. Future employment opportunities
48. Reputation of Institution
49. Work schedule
50. Flexible pacing for completing a program
51. Convenience
52. Distance from campus
53. Program requirements
54. Recommendations from employer
Students rank
each factor on
how important it
is to them.
Most Important Factors to Enroll Online
46. Financialassistanceavailable
47. Future employment opportunities
48. Reputation of Institution
49. Work schedule
50. Flexible pacing for completing a program
51. Convenience
52. Distance from campus
53. Program requirements
54. Recommendations from employer
These 3 factors
always rank as
the most
important.
Flexibility
•#1 Reason why student take online courses
•Not distance, not cost, not preferred
learning modality
TIME FLEXIBILITY
3. Windows of
Opportunity
What is a Window of
Opportunity?
The amount of time that an online
student is given to complete a
task; such as
• take a quiz,
• complete and submit an
assignment, or
• post to a discussion
• et cetera Item
#3
Example: Required Writing Assignment
• Opens Friday morning and closes Sunday night.
• Three days (but only two sleeps) might not seem too short for some people, but it
certainly would for other people,depending on their circumstances.
Piece of cake.
That’s perfect
for me.
No way can I
get that done
in time
One Big
Window?
What about one big window?
Opens on first day
Closes on last day
Barry’s
Thoughts on
Windows of
Opportunity
•Goldilocks is curious
•Common sense
helps
•Consistency is very
important
Item
#3
4. Student-
friendly Flexible
Design
Course design choices
made for online courses
tend to fall into one of
these categories:
1. Student-friendly
2. Faculty-friendly
3. Both
Item
#4
Example of “Friendliness”
• Instructor 1 isgoing to post a schedule for
weekly online office hours.
Step 1: instructor looks at hisweekly schedule
to see what will fit in
Step 2: chooses days/times that fit nicely for
him
Step 3: posts the schedule
• Instructor 2 isgoing to post a schedule for
weekly online office hours.
Step 1: instructor surveys students to see
which days and times they are availableto
meet, if needed
Step 2: chooses days/times that work best for
the most students
Step 3: posts the schedule
Other Design Considerations for Student-Friendliness
Beyond Design:
Instructional Procedures that are Student-Friendly
•Again,if the focus is placedsquarelyon the coursedesign,
variousthings will slip through the cracks.
•This next exampleis a caseof instructor behaviors(practices)
and it is unlikelythat someonewould design/planthe coursein
this manner.
Example of Untimely Feedback
Monday – Module 1 Essay Questions due date
Thursday – Module 1 Exam due date
Friday – Feedback on Essay Questions received by students
Timely Instructor Feedback
•Timely and useful feedback is needed to stay ontrack in online courses.
•Planto set asidesufficientgradingtime soon afterthe assessmentis due.
•Some facultyfind thatthey save time by providingaudioorvideo
feedbackrather than text, whichcan alsoincreaseyour instructorpresence
withonline students.
Barry’s
Thoughts on
Student-
Friendly
Design
• Being student-friendly does
not require you to be faculty-
unfriendly.
• Synchronous activity
alternatives?
• Form student groups based
on availability?
• Give option of not working
on weekends or holidays.
Item
#4
5. Continuous
Quality
Improvement
(CQI)
Not “one and done”
Iterations on quality
improvement – don’t eat
the whole elephant.
In and of itself, a course
quality rubric is a very
small cog in an overall
CQI process. Item
#5
Barry’s
Thoughts on
Continuous
Quality
Improvement
Some of the important things you
won’t find in a Course Quality
Rubric:
• Institutional support of DE
• Technology support of DE
• Instructional design support
• Online course development
support
• Ongoing PD for online
instructors
• Robust online student services
• Ongoing research and a culture
of evidence
Item
#5
Putting it All Together
Continuous
Improvement
Student
Friendly
Windows
Learning
Achieve-
ment
Instruction
Course
Design

More Related Content

Similar to Five Important Things You Won't Find in a Course Quality Rubric - Barry Dahl

OPID Conference Presentation
OPID Conference PresentationOPID Conference Presentation
OPID Conference Presentation
George Krueger
 

Similar to Five Important Things You Won't Find in a Course Quality Rubric - Barry Dahl (20)

Evaluating the Quality of Online Teaching
Evaluating the Quality of Online Teaching Evaluating the Quality of Online Teaching
Evaluating the Quality of Online Teaching
 
GVSU Blended Teaching and Learning
GVSU Blended Teaching and LearningGVSU Blended Teaching and Learning
GVSU Blended Teaching and Learning
 
Designing Online Courses for Flexibility
Designing Online Courses for FlexibilityDesigning Online Courses for Flexibility
Designing Online Courses for Flexibility
 
OPID Conference Presentation
OPID Conference PresentationOPID Conference Presentation
OPID Conference Presentation
 
Tamu module 2 faculty new
Tamu module 2 faculty newTamu module 2 faculty new
Tamu module 2 faculty new
 
Blended teaching and learning
Blended teaching and learningBlended teaching and learning
Blended teaching and learning
 
Best practices for teaching online
Best practices for teaching onlineBest practices for teaching online
Best practices for teaching online
 
Flipped Learning CTD
Flipped Learning CTDFlipped Learning CTD
Flipped Learning CTD
 
Online assessment
Online assessmentOnline assessment
Online assessment
 
Asynchronous 3 module model
Asynchronous 3 module modelAsynchronous 3 module model
Asynchronous 3 module model
 
Getting started with blended, a presentation for NMSU
Getting started with blended, a presentation for NMSUGetting started with blended, a presentation for NMSU
Getting started with blended, a presentation for NMSU
 
Making Connections - Students' Use of Past Feedback
Making Connections - Students' Use of Past FeedbackMaking Connections - Students' Use of Past Feedback
Making Connections - Students' Use of Past Feedback
 
Making Connections: How Students Use Feedback to Inform Future Learning
Making Connections: How Students Use Feedback to Inform Future LearningMaking Connections: How Students Use Feedback to Inform Future Learning
Making Connections: How Students Use Feedback to Inform Future Learning
 
D2L Community Webinar - Setting Online Faculty Expectations
D2L Community Webinar - Setting Online Faculty ExpectationsD2L Community Webinar - Setting Online Faculty Expectations
D2L Community Webinar - Setting Online Faculty Expectations
 
PRM 101 Session 1
PRM 101 Session 1PRM 101 Session 1
PRM 101 Session 1
 
Introducing the PASS program: Pre-arrival success strategies for students
Introducing the PASS program: Pre-arrival success strategies for studentsIntroducing the PASS program: Pre-arrival success strategies for students
Introducing the PASS program: Pre-arrival success strategies for students
 
Hybrid Courses & Curriculum
Hybrid Courses & CurriculumHybrid Courses & Curriculum
Hybrid Courses & Curriculum
 
CEC Module 5- Challenges of online teaching
CEC Module 5- Challenges of online teachingCEC Module 5- Challenges of online teaching
CEC Module 5- Challenges of online teaching
 
Needs assessment
Needs assessmentNeeds assessment
Needs assessment
 
UBD Presentation for schools and application.pptx
UBD Presentation for schools and application.pptxUBD Presentation for schools and application.pptx
UBD Presentation for schools and application.pptx
 

More from D2L Barry

More from D2L Barry (20)

Ask Not What AI Can Do For You - Nov 2023 - Slideshare.pptx
Ask Not What AI Can Do For You - Nov 2023 - Slideshare.pptxAsk Not What AI Can Do For You - Nov 2023 - Slideshare.pptx
Ask Not What AI Can Do For You - Nov 2023 - Slideshare.pptx
 
Designing Competency Structures and Learning Objectives
Designing Competency Structures and Learning ObjectivesDesigning Competency Structures and Learning Objectives
Designing Competency Structures and Learning Objectives
 
AI and ChatGPT in Online Education
AI and ChatGPT in Online Education AI and ChatGPT in Online Education
AI and ChatGPT in Online Education
 
Custom Pathways Resources - Kristin Randles.pdf
Custom Pathways Resources - Kristin Randles.pdfCustom Pathways Resources - Kristin Randles.pdf
Custom Pathways Resources - Kristin Randles.pdf
 
Don't Tell Me - Denise Huff.pptx
Don't Tell Me - Denise Huff.pptxDon't Tell Me - Denise Huff.pptx
Don't Tell Me - Denise Huff.pptx
 
Increase Learner Engagement and Cultivate Persistence in Brightspace - Dan Semi
Increase Learner Engagement and Cultivate Persistence in Brightspace - Dan SemiIncrease Learner Engagement and Cultivate Persistence in Brightspace - Dan Semi
Increase Learner Engagement and Cultivate Persistence in Brightspace - Dan Semi
 
Brightspace Creator +, Content Creation Platform for Engaging Interactives an...
Brightspace Creator +, Content Creation Platform for Engaging Interactives an...Brightspace Creator +, Content Creation Platform for Engaging Interactives an...
Brightspace Creator +, Content Creation Platform for Engaging Interactives an...
 
ABCs of Brightspace Awards - Dan Semi
ABCs of Brightspace Awards - Dan SemiABCs of Brightspace Awards - Dan Semi
ABCs of Brightspace Awards - Dan Semi
 
Brightspace Quick Eval Best Practices - Michelle Piper
Brightspace Quick Eval Best Practices - Michelle PiperBrightspace Quick Eval Best Practices - Michelle Piper
Brightspace Quick Eval Best Practices - Michelle Piper
 
10 Brightspace Tools You Didn’t Know About - Michelle Piper
10 Brightspace Tools You Didn’t Know About - Michelle Piper10 Brightspace Tools You Didn’t Know About - Michelle Piper
10 Brightspace Tools You Didn’t Know About - Michelle Piper
 
Ten Bright Ideas for Improving Accessibility in Online Courses - Barry Dahl -...
Ten Bright Ideas for Improving Accessibility in Online Courses - Barry Dahl -...Ten Bright Ideas for Improving Accessibility in Online Courses - Barry Dahl -...
Ten Bright Ideas for Improving Accessibility in Online Courses - Barry Dahl -...
 
E-Learning Mythbusters Revisited - ITC 2022.pptx
E-Learning Mythbusters Revisited - ITC 2022.pptxE-Learning Mythbusters Revisited - ITC 2022.pptx
E-Learning Mythbusters Revisited - ITC 2022.pptx
 
Office Documents: Making Word™ and PowerPoint™ Docs Accessible
Office Documents: Making Word™ and PowerPoint™ Docs AccessibleOffice Documents: Making Word™ and PowerPoint™ Docs Accessible
Office Documents: Making Word™ and PowerPoint™ Docs Accessible
 
Video Captions and Transcripts Made Easy , or at least easier
Video Captions and Transcripts Made Easy , or at least easierVideo Captions and Transcripts Made Easy , or at least easier
Video Captions and Transcripts Made Easy , or at least easier
 
Video Captions and Transcripts Made Easy, or at least easier
Video Captions and Transcripts Made Easy, or at least easierVideo Captions and Transcripts Made Easy, or at least easier
Video Captions and Transcripts Made Easy, or at least easier
 
Five Easy Wins for Making your PowerPoint Slideshows Accessible
Five Easy Wins for Making your PowerPoint Slideshows AccessibleFive Easy Wins for Making your PowerPoint Slideshows Accessible
Five Easy Wins for Making your PowerPoint Slideshows Accessible
 
D2L as a Training Platform for Faculty: Lessons Learned
D2L as a Training Platform for Faculty: Lessons LearnedD2L as a Training Platform for Faculty: Lessons Learned
D2L as a Training Platform for Faculty: Lessons Learned
 
Handout: YuJa, post to a discussion from a mobile device
Handout: YuJa, post to a discussion from a mobile deviceHandout: YuJa, post to a discussion from a mobile device
Handout: YuJa, post to a discussion from a mobile device
 
Handout: Channel Directions for YuJa
Handout: Channel Directions for YuJaHandout: Channel Directions for YuJa
Handout: Channel Directions for YuJa
 
Yuja the best, Don't just Yuja (D2L Connection Colorado)
Yuja the best, Don't just Yuja (D2L Connection Colorado)Yuja the best, Don't just Yuja (D2L Connection Colorado)
Yuja the best, Don't just Yuja (D2L Connection Colorado)
 

Recently uploaded

ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
ashishpaul799
 
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdfFinancial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
MinawBelay
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
中 央社
 
IATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdffIATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdff
17thcssbs2
 

Recently uploaded (20)

“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
An overview of the various scriptures in Hinduism
An overview of the various scriptures in HinduismAn overview of the various scriptures in Hinduism
An overview of the various scriptures in Hinduism
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjjStl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
 
How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdfFinancial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
 
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdfPost Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
 
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
 Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
 
The Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptxThe Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptx
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17
 
IATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdffIATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdff
 

Five Important Things You Won't Find in a Course Quality Rubric - Barry Dahl

  • 1. Barry Dahl Teaching & Learning Advocate, D2L Five Important Things You Won't Find in a Course Quality Rubric
  • 2. Brief History of Online Course Design Rubrics •2002: Cal State Chico developed the ROI (Rubric for Online Instruction) • Implemented at Chico in January 2003 •2003: FIPSE Grant won by Maryland Online: "Quality Matters: Inter-Institutional Quality Assurance in Online Learning“ • Grant term: Sept 2003 – Aug 2006
  • 3. A Brief Personal History • Feb. 2004 - QM workshop at ITC eLearning Conference • May 2004 - Faculty at my college voted to create a peer review process using V1 of QM rubric as the course design recommendations • 2005-present – former college still uses a peer review process with a rubric adapted from V1 of QM. • 2010 – began consulting gigs promoting the use of a three- pronged approach to improving online course and program quality • 2012 – began work at D2L – and here we are today
  • 4. 3 Major Components of e-Quality Learning Teaching Design
  • 5. Measuring the e-Quality Components Learning Teaching Design Learning Assessment Quality of Quality of Quality of
  • 6. Learning Teaching Design Learning Assessment Quality of Quality of Quality of We’ll leave this to everyone else The Other Presentations
  • 7. 5 Items for Today Learning Teaching Design Learning Assessment Item #2 Item #1 Item #3 Item #4 Item #5
  • 9. 1. Online Instructor Evaluation To determine the “quality” of the instruction provided, an evaluation needs to be conducted. Item #1
  • 10. Purposes of Instructor Evaluation Encourage excellence in teaching and learning. 01 Facilitate long- term quality improvement by monitoring instructional performance. 02 Provide constructive feedback by identifying strengths and areas for improvement. 03 Inspire professional growth and development. 04
  • 11. Faculty Evaluation Components • Student Evaluation of Instruction • Faculty Self-Evaluation • Professional Development Plan • Faculty Peer Review • Faculty Portfolio • Supervisor’s Evaluation of Faculty Typically some combination of the above components
  • 12. Barry’s Thoughts on Online Instructor Evaluation • It’s not the same • Classroom observations in traditional classroom are mostly for show • Observers/Evaluators need to “get” the art of online teaching • It CAN be done. Item #1
  • 13. Quality (Level) of Learning Achieved Learning Teaching Design Item #2 Learning
  • 14. 2. Online Student Collegewide Outcomes Assessment Similar achievement as F2F? Are online-only students included in your collegewide assessments? Have your accreditors asked to see this? Item #2
  • 15. Examples of Collegewide Assessments •Collegewide Outcome #1 • Written and oral communication • Practiced extensively, across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance • Assessment Projects for #1 • Assessments of writing skills • Scored by a cross- disciplinary team of faculty • Administered to random selection of students from across the college • Segregate writing samples of online students for comparison to on-campus students
  • 16. 2. Course Outcomes Assessment Comparisons Similar achievement as F2F? What can you learn from this? What to do about a lower performing group? Item #2
  • 17. Barry’s Thoughts on Quality of Learning •Culture of Evidence •Answer the skeptics •Don’t say “we can’t” •College-wide assessments for accreditors Item #2
  • 18. Why Do Students Take Online Courses? Caveat: assuming they aren’t forced to by a worldwide pandemic or something similar…
  • 19. Ruffalo Noel-Levitz PSOL •This survey can tell you how satisfiedyour students are and what issues are reallyimportantto them in the areas of: •Academic services •Enrollmentservices •Institutionalperceptions •Instructionalservices •Studentservices
  • 20. PSOL Factors to Enroll in Online 46. Financialassistanceavailable 47. Future employment opportunities 48. Reputation of Institution 49. Work schedule 50. Flexible pacing for completing a program 51. Convenience 52. Distance from campus 53. Program requirements 54. Recommendations from employer Students rank each factor on how important it is to them.
  • 21. Most Important Factors to Enroll Online 46. Financialassistanceavailable 47. Future employment opportunities 48. Reputation of Institution 49. Work schedule 50. Flexible pacing for completing a program 51. Convenience 52. Distance from campus 53. Program requirements 54. Recommendations from employer These 3 factors always rank as the most important.
  • 22. Flexibility •#1 Reason why student take online courses •Not distance, not cost, not preferred learning modality TIME FLEXIBILITY
  • 23. 3. Windows of Opportunity What is a Window of Opportunity? The amount of time that an online student is given to complete a task; such as • take a quiz, • complete and submit an assignment, or • post to a discussion • et cetera Item #3
  • 24. Example: Required Writing Assignment • Opens Friday morning and closes Sunday night. • Three days (but only two sleeps) might not seem too short for some people, but it certainly would for other people,depending on their circumstances. Piece of cake. That’s perfect for me. No way can I get that done in time
  • 25. One Big Window? What about one big window? Opens on first day Closes on last day
  • 26. Barry’s Thoughts on Windows of Opportunity •Goldilocks is curious •Common sense helps •Consistency is very important Item #3
  • 27. 4. Student- friendly Flexible Design Course design choices made for online courses tend to fall into one of these categories: 1. Student-friendly 2. Faculty-friendly 3. Both Item #4
  • 28. Example of “Friendliness” • Instructor 1 isgoing to post a schedule for weekly online office hours. Step 1: instructor looks at hisweekly schedule to see what will fit in Step 2: chooses days/times that fit nicely for him Step 3: posts the schedule • Instructor 2 isgoing to post a schedule for weekly online office hours. Step 1: instructor surveys students to see which days and times they are availableto meet, if needed Step 2: chooses days/times that work best for the most students Step 3: posts the schedule
  • 29. Other Design Considerations for Student-Friendliness
  • 30. Beyond Design: Instructional Procedures that are Student-Friendly •Again,if the focus is placedsquarelyon the coursedesign, variousthings will slip through the cracks. •This next exampleis a caseof instructor behaviors(practices) and it is unlikelythat someonewould design/planthe coursein this manner.
  • 31. Example of Untimely Feedback Monday – Module 1 Essay Questions due date Thursday – Module 1 Exam due date Friday – Feedback on Essay Questions received by students
  • 32. Timely Instructor Feedback •Timely and useful feedback is needed to stay ontrack in online courses. •Planto set asidesufficientgradingtime soon afterthe assessmentis due. •Some facultyfind thatthey save time by providingaudioorvideo feedbackrather than text, whichcan alsoincreaseyour instructorpresence withonline students.
  • 33. Barry’s Thoughts on Student- Friendly Design • Being student-friendly does not require you to be faculty- unfriendly. • Synchronous activity alternatives? • Form student groups based on availability? • Give option of not working on weekends or holidays. Item #4
  • 34. 5. Continuous Quality Improvement (CQI) Not “one and done” Iterations on quality improvement – don’t eat the whole elephant. In and of itself, a course quality rubric is a very small cog in an overall CQI process. Item #5
  • 35. Barry’s Thoughts on Continuous Quality Improvement Some of the important things you won’t find in a Course Quality Rubric: • Institutional support of DE • Technology support of DE • Instructional design support • Online course development support • Ongoing PD for online instructors • Robust online student services • Ongoing research and a culture of evidence Item #5
  • 36. Putting it All Together Continuous Improvement Student Friendly Windows Learning Achieve- ment Instruction Course Design

Editor's Notes

  1. Hello everyone. Thanks for joining us today  Many of you know me but for those of you who don't, my name is Barry Dahl, and I serve as the Teaching and Learning Advocate at D2L. After spending 27 years working inside Higher Ed, I've spent the last 10 years working with D2L primarily providing professional development opportunities for faculty and support staff who use D2L.  I also had the pleasure of serving on the Board of Directors of the Instructional Technology Council for a few years while still working at a college in Minnesota. This is an organization that's very near and dear to my heart. Let's get started with the next session titled: 5 important things you won't find in of course quality rubric. 
  2. Let's start with a brief history of course design rubrics for online courses. The first one that I know of and can find in the research was from 2002, the Cal State Chico rubric for online instruction or Roi. This preceded the quality matters rubric by about a year or more. Quality Matters began in 2003 when they were awarded a substantial FIPSE Grant which covered a three-year term ending in 2006. There are many other course design rubrics as well, but these two seem to encapsulate the start of the rubrics era
  3. My own history with course design rubric started in February 2004 when I attended the ITC eLearning conference and the people from Maryland Online were doing a workshop about Quality Matters. Because it was funded by a FIPSE Grant, they needed to share their developments during that Grant period, and they did share version one of their quality matters rubric with the audience at the conference. That same spring, I brought the rubric back to my College in Minnesota and with a large group of Faculty members we started working on our own process based on an adaptation of the quality matters rubric to create an online course peer review process. I'm pleased to say that 18 years later that process is still in place and has only grown in quality and acceptance over the years, or so they tell me. I also spent a fair amount of time working with other colleges as a consultant to take a look at their online course development procedures as well as accreditation efforts for online programs. In 2012 I started at D2L, and no longer work directly with course quality improvement processes, but I help direct the Course Quality Improvement Affinity Network at our annual Fusion conference and throughout the year.
  4. When I was doing my consulting work, this diagram got a lot of use. Once upon a time I called this my pyramid of e-Quality, and a faculty member quickly pointed out that it’s 2D, so it’s not a pyramid. So, I didn’t call it that anymore.  Part of my message was that there were at minimum three major components of e-learning quality. As you can see here in this diagram the three different colored triangles represent the quality of learning, of teaching, and of design. I guess you could say that the overall triangle represents the totality of e-learning quality – but that would only be true if lots of other factors were living in that white space in the middle.
  5. My contention then and it's still my contention today is that all three of these components are important when considering the quality of online  courses. The three components can be measured in many ways, but the most common ways are shown on the slide through assessments we measure the level of learning achieved, through some sort of performance evaluation process, we measure the level of teaching that occurred, and through things such as quality matters rubrics and similar instruments we measure the quality of online course design.
  6. Plenty of people do presentations on the course design rubrics, so we are not going to be talking about that today. So, what are we talking about? 
  7. We are going to be looking at 1) the quality of teaching, 2) the quality of learning, and then three other items that are not just tangential to the quality of an online course, but I would argue that they are central to it. 
  8. Let’s start with the quality of instruction. One of the issues with relying on a course design rubric is that although it looks at how the course is designed to operate, it is typically evaluated looking at an inactive course. You can't (typically) look at what is actually happening while the course is being taught. In other words, it doesn’t look at how well the course is taught.  For example, a course design rubric typically includes standards related to interaction, such as interactions between students and instructors. However, designing for interaction doesn’t answer the question about whether those interactions actually occurred and at what level they occurred and what value was achieved.
  9. Online Instructor Evaluations are critical to the overall e-Quality process, but they are not without a few potential hurdles or issues.  What are some of the potential issues with conducting an instructor evaluation to determine the “quality” of the instruction? Here are a few of them: Many institutions do not have an evaluation instrument appropriate for online teaching Many institutional leaders do not have the experience with online teaching & learning to be effective evaluators. Although classroom observations have a long history and are generally accepted practice, observations of the online classroom do not and are not. However, all the items above have been improving over the past 20 years.
  10. What are the goals or purposes of instructor evaluations? Things such as: Encouraging excellence Monitoring instructional performance Providing constructive feedback Inspiring professional growth
  11. Faculty evaluations typically are made up of some combination of the following: Student Evaluation of Instruction Faculty Self-Evaluation Professional Development Plan Faculty Peer Review Faculty Portfolio Supervisor’s Evaluation of Faculty
  12. Some of my thoughts about this process: Evaluating people who teach online is not the same as evaluating people who teach on-campus (of course many people do both) Classroom observations in traditional classroom are mostly for show, with the dean or other supervisor sitting in the back of the classroom observing some or all of a single class period. In online classroom, observation can be very informative and instructive, particularly if the Observers/Evaluators are well-versed in the art of online teaching Some schools have 20-25 years of experience with this process – so it CAN be done. (I used to often hear the objection of “we just can’t do it!”
  13. Let’s move on to the quality or level of learning that happens in online classes. There’s really two lenses that I think need to be examined here. Course-level outcomes and college-wide outcomes. We have a problem if you can’t provide evidence of students successfully completing course outcomes. We also have a more general problem if online students overall are less successful than F2F students in achieving college-wide outcomes.
  14. Let’s start with assessments of collegewide and Program-level learning outcomes. Do your online students achieve the Collegewide Outcomes and/or Program Outcomes at similar levels as F2F students? Are you even including (and segregating) online-only students in your collegewide assessments? Have you been asked by your accrediting bodies for this information? Or have you reported on this info to your accreditors as part of a review process?
  15. An example from my former college Collegewide Outcome #1 Written and oral communication Practiced extensively, across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance Assessment Projects for #1 Assessments of writing skills Scored by a cross-disciplinary team of faculty Administered to random selection of students from across the college Segregate writing samples of online students for comparison to on-campus students
  16. And what about course-level learning outcomes. Do your online students achieve the expected course Outcomes at similar levels as F2F students in other sections of the same course? Are you tracking Competencies with different groups of learners and comparing? What can you learn from this? One thing is you can discern your own slice of the pie of the “No Significant Difference” phenomenon and possibly drill into smaller differences in the courses you teach. What to do about a lower performing group? Possibly remediation for one group or the other. Redesign activities and assessments to help the lower achieving group be more successful
  17. Develop a Culture of Evidence, and this will be part of it. Not sure if you’ve noticed, but there are still, in 2022, there are still skeptics of online learning. Be prepared to answer them with data. If you say “but we can’t do that with online students…” then we have a bigger problem to deal with This was a big deal 15-20 years ago when getting online programs approved by regional accreditors. It still might be a big deal for your accreditor.
  18. Let’s examine this question: Why Do Students Take Online Courses? How do you answer this question? Why do students take online courses? in normal times, making normal decisions for oneself? In other words, not during a pandemic when remote (online) learning was the only option. I am talking about why students take online courses when they have choices related to the course delivery method.
  19. Many of you are probably familiar with the PSOL, which is the priorities survey of online Learners from Noel Levitz. Back in my college vice-president days, we used the survey 5 times between the years of 2003 and 2009 to gather data about student satisfaction as well as some other factors related to their online learning experiences at the college. I have continued to follow the results of this annual survey since then – partly because I’m a bit of a data nerd – but also because this survey is the one place where I learned the most about our online students and especially how our survey results compared to others who used the survey. The crux of the survey is satisfaction based, looking at how satisfied online students are with various different services at the college and the quality of instruction received 
  20. There's also another portion of the survey that asks about the factors or reasons why they chose to enroll in online courses. You can see the list of nine items on the screen. Three of those nine items consistently have much higher importance levels than all the other items. Which three do you believe would be at the top of their lists? 
  21. If you guess the three items in the middle, you are correct. Work schedule, flexible pacing, and convenience, consistently rank as the most important factors.  Almost all of the data that I have seen, both organization specific as well as Nationwide data, point to these three items being by far the most important factors that drive student enrollment in online courses. This is especially true at the undergraduate level. so looking at work schedule, flexible pacing, and convenience, we see a common thread
  22. It seems perfectly logical to me to look at those three items and their similarities and come up with the number one reason why students take online courses. We talk about distance education but it's not really distance. It's not the cost of education because sometimes it's higher than face to face education, in fact usually it is higher. The idea of preferred learning modality is kind of a bunch of hooey.  Instead, those things all relate to the students having time flexibility in their lives. The ability to fit an education into their otherwise full schedules and busy lives. 
  23. So, the third factor on our list of five factors today has to do with windows of opportunity. Again this is not something you'll find in a course design rubric, but it's incredibly important for students to manage their time restraints related to coursework.  A window of opportunity is the amount of time that an online student is given to complete a particular task or activity or a series of actions. Faculty members usually make the call on how long the window opportunity will be for each item, which makes sense since they are the ones designing the activities but ensuring that the students have a large enough window of opportunity is a key for student success. 
  24. Here's a very simple example, not meant to insult anybody's intelligence, but just to make sure we illustrate what I'm talking about. A writing assignment that the professor posts on Friday morning and the due date is Sunday night so students have three days, in other words, the weekend. For one student that might be awesome. Plenty of time to get it done. For another student that may be incredibly short and they're going to have to disrupt their lives in order to meet the due date.  And what about weekends? 
  25. Once upon a time, I took the advice of two students and tried having just one big window for a 16-week Personal Finance course. Everything was available on Day one and everything was due on the last day. There are several reasons that this is not a great idea, unless you’re shooting for having an online correspondence course with very little student interaction – not what most of us are shooting for.   The result was that it worked great for two or three students, and not so great for everybody else. 16 weeks to get the work done, and many students waited until week 14 or 15 to get serious about doing the work. Procrastination is the king of the world. 
  26. What would Goldilocks say? How do you know if they’re too long, too short, or just right?  We could make up an answer…but…Common sense goes a long way Big projects need bigger windows Student situations are not homogenous – it’s not one size fits all Consistency is very helpful to online students. Keep windows similar for similar tasks.
  27. Generally speaking (with exceptions), design choices that are made for online courses tend to fall into one of these categories: Student-friendly Faculty-friendly Both
  28. Here’s a simple example Instructor 1 is going to post a schedule for weekly online office hours. Step 1: instructor looks at his weekly schedule to see what will fit in Step 2: chooses days/times that fit nicely for him Step 3: posts the schedule Instructor 2 is going to post a schedule for weekly online office hours. Step 1: instructor surveys students to see which days and times they are available to meet, if needed Step 2: chooses days/times that work best for the most students Step 3: posts the schedule Instructor 1 is faculty-friendly. Instructor 2 is student-friendly.
  29. Other design considerations to make your course student-friendly Days and times of the week when due dates are scheduled Required synchronous activities Required group work Modes of contact for online office hours; both the timing of instructor availability and the mode/methods of contact. Appropriate amount of course content Extra credit or make-up opportunities for required coursework that is missed
  30. Looking beyond the course design. What about some instructional practices that are Student-Friendly?  Again, if the focus is placed squarely on the course design, various things will slip through the cracks. This next example is a case of instructor behaviors (practices) and it is unlikely that someone would design/plan the course in this manner.
  31. This is an example of untimely feedback from the instructor to the student.   On Monday the student is supposed to turn in essay question answers for module one. The module one exam is on Thursday of that same week, but the instructor feedback on the answers to the essay questions doesn't get delivered to the students until Friday after the exam.   Again, it's unlikely that someone would design the course with this in mind, but reality doesn't always match design. 
  32. Instructor Feedback is one of the items that the PSOL survey shows is super important to students but with which they typically have a low satisfaction rate. Timely and useful feedback from instructors is important for keeping students on track in online courses. Plan to set aside sufficient grading time soon after the assessment is due. Some faculty find that they save time by providing audio or video feedback rather than text, which can also increase your instructor presence with online students.
  33. Being student-friendly does not require you to be faculty-unfriendly. If synchronous activities are required, have you provided opportunities for students who cannot be in a certain place at a certain time?  If group work is required, have you provided opportunities for students to form groups based on availability?  Are your due dates scheduled in a way, in combination with the length of the windows of opportunity, that provide students with the option of not working on weekends or holidays?
  34. The fifth item was to do with the concept of CQI, or Continuous Quality Improvement. Don’t treat your Quality Process as a “one and done” operation. By making it a continuous process, you can make progress over time rather than trying to achieve greatness all at once. In and of itself, a course quality rubric is a very small cog in an overall CQI process.
  35. Some closing thoughts about Continuous Quality Improvement and the shortcomings of course design rubrics Some of the important things you won’t find in a Course Quality Rubric: Institutional support of online education Technology support for online education Instructional design support Online course development support Ongoing professional development for online instructors Robust student services for online students Ongoing research and a culture of evidence
  36. Now comes the easy part. Just take all these moving pieces and create yourself a well-oiled machine of e-Learning greatness.