Underlying Principles of QMQM is a faculty-driven, peer review process that is… Collaborative Collegial Continuous Centered - in academic foundation - around student learning
For Our Purposes, Quality Is…• More than average; more than “good enough”• Attempt to capture what’s expected in an effective online course at about an 85% level• Based on research and widely accepted standards
Factors Affecting Course Quality QM Reviews Course Design ONLY
Design vs. Delivery The faculty member is integral to both design and delivery.Course Design … Course Delivery …is the forethought and is the actual teaching of theplanning that a faculty course, the implementation ofmember puts into the course. the design.QM is about DESIGN - not delivery or faculty performance
Distinguish between design and delivery… Example: Discussion BoardDesign: Delivery:Discussion board How often the facultyplanned in course; member actuallystudents told how they participates in theshould participate and discussion; what thehow they can expect the faculty member actuallyfaculty to participate. says to students.
About The QM Rubric 2011-2013Eight General Standards: 1. Course Overview and Introduction 2. Learning Objectives (Competencies) 3. Assessment and Measurement 4. Instructional Materials Key components must align. 5. Learner Interaction & Engagement 6. Course Technology 7. Learner Support 8. Accessibility Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.
Course Learning Objectives (2) Learner Interactions & Activities (5) Resources, Materials (4) & Technology (6) Assessment and Measurement (3)Key Sections that Must Align
What QM is NOT About…• Not about an individual instructor (it’s about the course)• Not about faculty evaluation (it’s about course quality)• Not about judgment (it’s about diagnosis and improvement)• Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)
QM = Process and Rubric Process RubricOFFICIAL Outcome: Earn QM recognition Outcome: Earn QM recognition Must follow official QM guidelines Tool to assess online courses during and procedures formal QM review Outcome: Improve courses, Outcome: Improve courses, meet meet institutional goals institutional goals, demonstrate commitment to quality Guide to develop new online courses AND review and update online coursesINFORMAL Outcome: Improve courses, meet Outcome: Improve courses, meet institutional goals, demonstrate institutional goals, demonstrate commitment to quality commitment to quality Institutions determine use and Guide to develop new online procedures courses AND review and update online courses
About the Course• QM is designed to review “mature” courses (taught at least two semesters)• QM logo indicates year course met expectations• Triggers for subsequent reviews: – Faculty request – More than 3 years since original review – New textbook or instructor – Professional or accreditation review pending
The Peer Review Team• 3 faculty peer reviewers: • must be experienced online instructors • must attend QM training • one MUST be external to the course’s originating institution • there must be a subject matter expert (SME) on the team. NOTE: The SME could also be the external reviewer. AND• Faculty course developer: • access to rubric prior to review • involved in pre-review discussions • consulted during review
About the Review• On average, a course review takes 7-10 hours• Factors affecting review time include • Reviewer familiarity with the discipline • Reviewer familiarity with the LMS • Reviewer familiarity with the QM review process • Organization of the course
Review Outcome• If meets expectations and is recognized: – Recognized by Quality Matters – Notifications distributed – ID support provided if requested and available – Recognized courses on QM website• If does not yet meet expectations: – Instructor (and/or ID) make changes – Team Chair/Master Reviewer approve revisions – Course meets expectations and is recognized
Rubric Scoring# Standards Points Relative Value Total Points 21 3 Essential 63 12 2 Very Important 24 8 1 Important 8 41 95
In an Official QM Review• Team of 3 reviewers initially score individually• One score per standard based on team majority• Pre-assigned point value• Met/Not Met decision; All/None points• Consensus is NOT required
Points are NOT on assigned on a sliding scale…Standards If the standard If the standard is met … is not met …Essential 3 points 0 pointsVery 2 points 0 pointsImportantImportant 1 point 0 points
To Meet Expectations…Two ThresholdsA course must achieve: “Met” on all 21 of the 3-point “essential” standards.1. A minimum of 81 out of 95 points (81/95 = 85%)2.
The 85% RuleTwo 85% Uses:A minimum of 81 out of 95 points(81/95 = 85%)Reviewers use their own gauge of85% as to if a standard is met or not.Standards do not have to be 100% tobe marked “Met.”
Key Sections That Must Align Course Learning Objectives (2) Learner Interactions & Activities (5) Resources, Materials (4) & Technology (6) Assessment and Measurement (3)
Think About “Alignment”Objectives From Intro Assignment: Read Chapter 4 (assume it deals withPsych relationships between biology and human behavior) in text; review study guide, objective 2, page 4.Course : Summarize Assignment: View Chapter 4 PPT; listen to Chapter 4the relationships that podcastexist between biology Graded Assignment:and human behavior Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specificModule ways this area affects human behavior.1.Identify and define 7 Graded Discussion Forum Question: Psychology is themajor biological areas. study of behavior and mental processes. Why, then, are2.Recognize examples psychologists interested in biological matters such as theof how each area nervous system, the endocrine system and heredity?affects behavior. Read and respond to classmates’ postings. Exam: 50-item multiple choice exam taken in campus Testing Center.
General Standard 1 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 1, The course introduction You must share your deepCourse Overview and sets the tone for the knowledge of courseIntroduction: course, tells students what 1.Organization to expect, and provides 2.NavigationThe overall design of the guidance to ensure they get 3.Expectations (Policies)course is made clear to the off to a good start.student at the beginning ofthe course.
Standard 1Answer the following questions:1.Do you know what to do first?2.Do you know how to start the course?3.Note your initial reaction to the course:4.Do you feel comfortable with the navigation?5.What are your first impressions about how this course works?
General Standard 2 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 2, The learning objectives You must provideLearning Objectives: establish a foundation upon measurable, precise which the rest of the course learning objectives at theLearning objectives are is based. course- and module-levelmeasurable and are stated objectives so yourclearly. students will know what is expected of them.
Standard 2 as a SentenceQuality ismeasurable objectives (2.1)and consistent module-level objectives (2.2),that are written from the student perspective (2.3),with instructions on how to meet them (2.4),that are appropriate for the level of the course (2.5). Submitted by Sasha Thackaberry, Cuyahoga Community College
Standards 2.1 & 2.2Write measurable course and module learningobjectives that•precisely describe what students will gain frominstruction•guide instructors to accurately assess studentaccomplishment.•are consistent
Measurable Learning Objectives• Begin with an action verb• Precisely describe behavior that can be observed or evaluated
Test of “Measurability”A Measurable Learning Objective should completethis sentence:Upon completion of this course/module,students will be able to (DO SOMETHING).Helpful Hundred….
Standard 2: Learning Objectives Bloom In Practice University of Victoria Counseling Serviceshttp://www.coun.uvic.ca/learning/exams/blooms- taxonomy.html
Analyze Learning Objectives• Complete the table (handout): Look at ANT 103 learning objectives, Apply Standards 2.1 through 2.5 Quality is measurable objectives (2.1) and consistent module-level objectives (2.2), that are written from the student perspective (2.3), with instructions on how to meet them (2.4), that are appropriate for the level of the course (2.5).
General Standard 4 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 4,Instructional Materials: The instructional materials You must provide rich, form the core of the course, robust and appropriateInstructional materials are and these standards respect resources and materialssufficiently comprehensive the instructor’s prerogative that support your statedto achieve stated course in selecting them. The focus learning objectives.objectives and learning of this standard is onoutcomes. supporting the course objectives and competencies, rather than on qualitative judgments about the materials.
Apply Standard 4.1Standard 4.1 Met Not Met Helpful RecommendationThe instructionalmaterialscontribute to theachievement of thestated course andmodule/unitlearning objectives.
General Standard 5 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 5,Learner Interaction & Engaging students to You must provideEngagement: become active learners meaningful and contributes to the learningForms of interaction process and to student productive activities thatincorporated in the course persistence. prompt your students tomotivate students and actively practice theirpromote learning. learning.
Standard 5.1Learning Objective Possible Aligned Activities 1.Analyze thecauses of theCivil War and 2.how theypersist in our 3.modernsociety.
Standard 5.2Type of InteractionStudent-InstructorStudent-ContentStudent-Student
General Standard 3 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 3,Assessment and Assessment is implemented Your assessments mustMeasurement: in a manner that not only align with your stated allows the instructor a broad learning objectives andAssessment strategies are perspective on the students’ guide students todesigned to evaluate mastery of the content, but measure their ownstudent progress by also allows students to learning progress.reference to stated learning measure their own learningobjectives; to measure the throughout the course.effectiveness of studentlearning; and to be integralto the learning process.
Apply Standard 3.1 using Learning Objective Standard 3.1 Met Not Met Helpful RecommendationThe types ofassessmentsselected measurethe stated learningobjectives and areconsistent withcourse activitiesand resources.
General Standard 6 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 6,Course Tools and The technology enabling the You must select the toolsTechnology: various course components and media that best facilitates the student’sCourse navigation and learning experience and is support your learningtechnology support student easy to use, rather than objectives and provideengagement and ensure impeding the student’s opportunities for youraccess to course progress. students to activelycomponents. practice their learning.
Standard 6 Includes…Course Tools –•functional software that provide areas for interaction inthe course•Examples include forums, quizzes, surveys, blogs, socialmedia, wikis, etc.Media –•one-way delivery modalities that enhance learning•Examples include YouTube videos, podcasts, audiorecordings, pictures, graphics, etc.
General Standard 7 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 7,Learner Support: In the learner support You must direct your standard, four different online students to theThe course facilitates kinds of support services are support services theystudent access to addressed: technical need to be successful andinstitutional support support, accessibility would be available toservices essential to student support, academic services them if they attendedsuccess. support, and student campus-based courses. services support.
General Standard 8 About the Brief Description To meet the General Standard from the Rubric General StandardGeneral Standard 8,Accessibility: The accessibility standard You must design your incorporates the principles course so students withThe course demonstrates a of Universal Design for special needs can accesscommitment to accessibilityfor all students. Learning (UDL) and is course materials and be consistent with Web successful in your course. Content Accessibility Guidelines (WCAG).
Sites to ExploreUniversal Design for Learning WCAGSite provides “a framework for •Web Content•designing educational Accessibility Guidelinesenvironments that enable all (WCAG)learners to gain knowledge, •Site providesskills, and enthusiasm for documents that “explainlearning how to make Web•reducing barriers to the content more accessiblecurriculum to people with•providing rich supports for disabilities.”learning.”
Standard 8.1Consider the following scenario:An online Nursing course is offered through Desire2Learn andrequires that students watch six videos that illustrate keyconcepts in critical care. These videos are provided by thetextbook publisher and are not closed-captioned.If you were reviewing this course, how you would applyStandard 8.1 and what helpful comments would you give theinstructor?
Professor Dancealot Needs Help!• Watch this brief You Tube video (You Tube: http://www.youtube.com/watch?v=1k8aeDUC9XQ)• Apply your understanding of the General Standards and alignment.• Convince the Professor that he needs to “practice alignment” if he wants to help his students meet the stated learning objectives of this course.
Spur Your ThinkingCourse Objectives Standard 3.1Upon completion ofthis course, you willbe able to : Standard 4.1:1. Perform the basicsteps of foxtrot, waltz,swing, chacha, rumbaand tango in proper Standard 5.1:dance position.2. Demonstrateconfidence on the Standard 6.1:dance floor.3. Find your eternaldance partner.
Peer Reviewer EligibilityHow Do I You must be ELIGIBLE:Become a •Must complete this F2F Applying the QM Rubric workshop.QM Peer •Must complete all assessments in theReviewer? online Peer Reviewer Course – Two weeks online – Includes review material – Objective and written assessments •Must have recent (within last 18 months) online teaching experience.
Register for Peer Reviewer Course Sign up for a Peer Review Course at http://qmprogram.orgPrint your Certificate of Completion at MyQM!
Underlying Principles of QM1. Based on national standards of best practice, the research literature and instructional design principles2. The course under review does not have to be “perfect” but better than just “good enough” (Standards met at about 85% level or better.)3. Integral to a continuous quality improvement process4. Valued as part of a faculty-driven, peer review process5. Designed to promote student learning
Underlying Principles of QM cont.6. Designed to ensure all reviewed courses will eventually meet expectations7. QM is a collegial review process, not an evaluation process8. A review team must include an external peer reviewer9. Set up so that the course faculty or instructor considered part of the review team
Who’s On the Team? 3 faculty peer reviewersFormal – Must attend QM Training – Must be experienced online instructorsPeer – One MUST be external to the course’s originating institutionReview – Must include one subject matter expert (SME) on theTeam team NOTE: The SME could also be the external reviewer – Chaired by Master Reviewer AND Faculty course developer (Instructor) – Access to rubric prior to review – Involved in pre-review discussions – Consulted during review
To Meet Expectations: Two ThresholdsA course must achieve: “Yes” on all 21 of the 3-point “essential” standards.1. A minimum of 81 out of 95 points2.
Online & Hybrid Courses• Rubric designed for both• Same set of standards apply to both• How standards achieved may differ• For blended courses, focus on pedagogical integration of online and onsite components
Your Point of View …As a QM Course Reviewer, you should:• Take the students’ point of view• Advocate for the student• Support your decisions with – Citations from the standards and annotations – Evidence from the course
Thanks to YOU…Quality Matters! More Information at: www.qmprogram.org