This document discusses ongoing efforts to improve the process for reviewing online course development and delivery at Ozarks Technical Community College. It provides background on how the college arrived at its current updated review process by drawing from various quality standards frameworks. The implementation involves conducting separate development and delivery reviews, piloting review documents, and soliciting feedback. Results so far indicate that the approach has led to consistent improvement in online courses over time and greater instructor engagement in professional development.

1. Online Course Development and
Delivery Reviews: An Ongoing
Process
Carla R. Bradley
OTC Online
Ozarks Technical Community College
bradleyc@otc.edu
https://twitter.com/carlabradley1

2. Words
Consistency standards
Coach Challenges
Guide
Effective practices

3. Words
• Evaluation • Peer review
• Evaluator • Coach
• Evaluator • Mentor
• Requirements • Effective practices
• Problems • Challenges
• Rubric • Guide
• Requirements • Consistency standards
• Requirements • Quality initiatives

4. Goals
1. Explain how OTC Online arrived at
updated online review process.
2. Address implementation strategies and
feedback.
3. Share hints for reviewers

5. Previous Online Review Document
Checklist and comments
Review document with “yes” or “no” that
combined development and delivery
Lacked extensive information on instructional
content, technology, academic integrity, and
accessibility.

6. Instructor Quality Results

8. Chasen, Michael. Blackboard World 2010.

9. Chasen, Michael. Blackboard World 2010.

10. 3 Resources
Quality Matters
Blackboard Exemplary Course
Chico State

11. Results of course review process

12. Blackboard
Quality Matters Exemplary Chico State
Course
1. Course overview 1. Course design 1. Learner support
and introduction 2. Interaction and resources
2. Learning and 2. Online
objectives organization and
collaboration design
3. Assessment and 3. Assessment 3. Instructional
measurement design and
4. Learner
4. Resources and delivery
support
materials 4. Assessment and
5. Learner evaluation of
engagement student learning
5. Innovative
6. Course teaching with
technology technology
7. Learner support 6. Faculty use of
8. Accessibility student feedback

13. Quality Matters
“The course review process is a collegial discussion
between faculty peers committed to Continuous
quality improvement. It is not an evaluation. Centered
in national standards of best practice, the research
literature and instructional design principles designed
to promote student learning. . . .as a diagnostic tool to
facilitate the continuous improvement of online
courses.”

14. QM Example
Striving for Quality in Online Courses- J.J. Johnson-Pierce
College in Washington State
Quality assurance team (of six):
Sensitive about academic freedom
Student feels comfortable- course layout
Anticipate questions
Student comfort (home page)- consistent interface
Measure of course design
Composed of instructors in diverse disciplines
Mentoring and coaching provided
Goal-
• Go to the next level and tie to PD sessions

15. Blackboard Exemplary Course
“The Blackboard Exemplary Course Program began
in 2000 with the goal of identifying and
disseminating best practices for designing engaging
online courses.”

16. Bb ECP Example
• Excellence within our Grasp: Using Blackboard’s
Exemplary Course Rubric- Mark Burris- University
of Arkansas Little Rock and Sue Burris- National Park
Community College
• Goal 1: Promote instructional excellence across
Arkansas through online student engagement
principles, moving beyond “buttonology”
• Goal 2: Promote academic collaboration through
local and statewide community of practice

17. Chico State Example
Online Course Assessment (James West)- Missouri
Baptist University
Peer evaluations results from spring 2005 to spring
2010
– High score has been consistent
– Low score has had an upward trend
– Mean has been upward
– Consistent improvement over 5-year period

19. More with Less
• More than 200 online instructors each semester
(and growing)
• Fall 2011 enrollment 15.9% since fall 2010
• Total of 4,709 students taking at least one online
class
• Funding

20. Growth of OTC Online

21. Online Development Review

22. Online Delivery Review

23. Unique to College
Navigation
Online syllabus template (current)
Instructor response time
Assessment data

24. Accessibility and Compliance
“Creates duplicate channels of information to benefit
all learners.”
Planning will have a cost-savings rather than
retrofitting.
Builds awareness.
Georgia Tech Research on Accessible Distance Education (GRADE). CATEA Access
E Learning. http://www.accesselearning.net/mod1/1_01.php

26. Pilot
1. Introduced in OTL 101 in fall 2010.
2. Fall 2010 online instructors completed.
3. Spring and summer 2011 online instructors
completed (some courses reviewed).
4. Fall 2011 only online instructors reviewed
completed.

27. Approach to Online PD
January 2010- present
427 participants (83 participated
in one or more sessions)
Move away from “buttonology”

28. Review Document
1. Separate development and delivery
2. Add pre-review comment section
3. Organize by navigation
4. Pilot review documents and solicit feedback
5. Involve departments
6. Check with DSS
7. Invite feedback from OTC Online Committee
8. Meet online with Online Review Task Force Team

29. Change is Tough

30. Feedback
“I really like the way the reviews are being
conducted. . . . I also like that instructors are given
the review sheets first: it gives us very clear
guidelines and allows us to see first in which areas
we can improve. On my one review, I used the
review sheet as a guideline to improve, but if I had
been in possession of the sheet much earlier, I
would have done much better in that first class.”

31. Unexpected Outcome

32. https://twitter.com/carlabradley1