Informational Webinar: Quality Online Learning and Teaching (QOLT)Brett Christie, Ph.D., CSU Academic Technology Services Li Wang, Ph.D., CSU Northridge Dora Preminger, Ph.D., CSU Northridge
BEST USE OF THE WEBINAR• If you haven’t done so already, enter your name, campus, and role in the chat box.• QOLT will be described in as much detail with as much clarity as possible. Visuals are included and all information is available at the QOLT site.• Feel free to enter comments/questions into the chat box. We will screen those and address all possible.• Should questions arise after the session, contact Brett Christie (firstname.lastname@example.org).• Session is being recorded for review or referral at QOLT site.
AGENDA• Introductions & Acknowledgments (5 minutes) • Brett Christie, CSU Faculty Development Liaison, QOLT Program Director • Li Wang, CSUN Instructional Designer, Quality Matters, Research Colleague, QOLT Associate • Kathy Fernandes & John Whitmer, CSU LMSS leadership• Overview of QOLT (10 minutes) • How we got here • Purpose; Brief review of 2011-2012 pilot • 2012-2013 Project timeline• A closer look at the QOLT instrument and process (20 minutes) • Instructor Self-review • Student Report • Campus Coordinator review/verification role• Discussion (20 minutes)• Closure and feedback (5 minutes)
QOLT PURPOSE• Create a useful evaluation tool that can help faculty (re)develop quality hybrid/online courses.• Identify exemplary practices for teaching and learning through hybrid/online courses.• Inform faculty development activities and programs related to hybrid/online teaching.• Recognize faculty, programs, and campuses that are creating quality online courses. And, share!
MODELS EXAMINED• Rubric for Online Instruction: Designed to assist development and evaluation of online courses while promoting dialog about the nature of student learning.• Quality Matters: Faculty-centered, peer-review process designed to certify quality of hybrid/online courses.• Quality Online Course Initiative: Course rubric and evaluation system developed in Illinois to improve online accountability.• Course Review Toolkit:An instrument and a system, developed at CSU Northridge, for informing and assessing Face-to-Face, Hybrid, and Online, teaching and learning.• Online Course Evaluation Program: Developed by the Monterey Institute for Technology and Education, OCEP is a criteria-based evaluation tool to assess and compare the quality of online courses.• eLearning Best Practices Rubric @ Sacramento State• Various research and publications on effective teaching, learning, design, assessment, syllabi, etc.
RESULTING QOLT EVALUATION INSTRUMENT1. Course Overview and Introduction (8)2. Assessment and Evaluation of Student Learning (6)3. Instructional Materials and Resources Utilized (6)4. Student Interaction and Community (8)5. Facilitation and Instruction (8)6. Technology for Teaching and Learning (6)7. Learner Support and Resources (3)8. Accessibility and Universal Design (4)9. Course Summary and Wrap-up (3)
• Participants should not enter into lightly• Participation time estimated at 1+ hour• Course evaluation, not simple survey• Process is about analysis and reflection• Pay-off is informed course redesign• Validation of teaching effectiveness effort
QOLT SCORING PER OBJECTIVEThere is also an open-ended box per section for descriptive.
Section 1. Course Overview and Introduction Instructor gives a thorough description of the course, as well as introducing students to the course protocol and expectations.
1.B. ‘COURSE DESCRIPTION’ BASELINE§ There should be specific meaning to the course,section, instructor, and students, not just theminimal course description that appears in thecatalog."Directed conversation in Spanish for elementary-levelstudents. Includes individual and class assignments inlaboratory. May be repeated for credit. Admission byconsent of instructor."
1.D. ‘ONLINE ETIQUETTE’ SAMPLE§ Do not dominate any discussion.§ Do not use offensive language.§ Never make fun of someone’s ability to read or write.§ Use simple English.§ Use correct spelling and grammar.§ Share tips with other students.§ Keep an “open-mind” and be willing to express even your minorityopinion.§ Be aware of the University’s Academic Honesty Policy.§ Think before you push the “Send” button.§ Do not hesitate to ask for feedback.§ When in doubt, always check with your instructor for clarification.Mintu-Wimsatt, A. (2010). Netiquette: Make it part of your syllabus. Journal of Online Learning andTeaching, 6(1),
1.F. ‘TECHNICAL COMPETENCIES’ SAMPLE§ Connect to the Web using a web browser.§ Navigate around the Web and use search engines.§ Send and receive e-mail with attachments.§ Basic word processing, including cutting and pasting.§ Open, save, and manage files.§ Organize folders and files (create, name, rename, move).§ Compress files and folders.*Depends on course level and proportion of online mode
SECTION 1, RUBRIC:“COURSE OVERVIEW AND INTRODUCTION”
SECTION 2: ASSESSMENT OF STUDENT LEARNING Student Evaluation and Assessment refers to the process used to gather evidence of the achievement of the Student Learning Objectives/Outcomes (SLOs). We strongly recommend that instructors contactthe Office of Academic Assessment (or similar) for assistance and information about this section.
Objectives Example2.a All Student Learning Objectives/Outcomes (SLOs) are If the mandated course level objectives are not measurable, thenspecific, well-defined, and measureable. module level objectives should be measurable and support course level objectives.2.b The grading policy is clearly stated for the course and Instructor provides late submission policy and scale, weights ofindividual assignments. respective assignments, and the corresponding letter grade if scores are accumulated at the end.2.c The learning activities (including the assignments and Instructors explain how learning activities such as assignments,ungraded activities) promote the achievement of the SLOs. discussions contribute to the achievement of the stated SLOs.2.d The assessment instruments selected are sequenced, There are multiple ways to demonstrate mastery-e.g., project, paper,varied, and appropriate to the student work being assessed. tests. One is built upon the other tool.2.e Throughout the semester Instructor provides multiple Activities may include but not limited to blogs for reflection, peeropportunities to give feedback on students learning (strengths review, practice test and draft of term paper, module summary.and weaknesses) and to “self-check” students learning/progress.2.f Throughout the semester, instructor provides multiple Instructor may consider the use of surveys, discussion forums, oropportunities to solicit feedback from their students about their item analyses to collect feedback or attitudinal data (that goeslearning and on the course for the improvement of the course. beyond student learning outcomes) on the effectiveness or difficulty of the resources and activities (e.g., “Muddiest Point”), or item analysis of test questions in order to improve the course in the future.
Objectives Example2.a All Student Learning Objectives/Outcomes (SLOs) are If the mandated course level objectives are not measurable, thenspecific, well-defined, and measureable. module level objectives should be measurable and support course level objectives.2.b The grading policy is clearly stated for the course and Instructor provides late submission policy and scale, weights ofindividual assignments. respective assignments, and the corresponding letter grade if scores are accumulated at the end.2.c The learning activities (including the assignments and Instructors explain how learning activities such as assignments,ungraded activities) promote the achievement of the SLOs. discussions contribute to the achievement of the stated SLOs.2.d The assessment instruments selected are sequenced, There are multiple ways to demonstrate mastery-e.g., project,varied, and appropriate to the student work being assessed. paper, tests. One is built upon the other tool.2.e Throughout the semester Instructor provides multiple Activities may include but not limited to blogs for reflection, peeropportunities to give feedback on students learning (strengths review, practice test and draft of term paper, module summary.and weaknesses) and to “self-check” students learning/progress.2.f Throughout the semester, instructor provides multiple Instructor may consider the use of surveys, discussion forums, oropportunities to solicit feedback from their students about item analyses to collect feedback or attitudinal data (that goestheir learning and on the course for the improvement of the beyond student learning outcomes) on the effectiveness orcourse. difficulty of the resources and activities (e.g., “Muddiest Point”), or item analysis of test questions in order to improve the course in the future.
Sample Online Assignment Rubrics• Online Discussions• Student/Group Wikis• Student Blogs• Twitter• PowerPoint or Enhanced PodCast• ePortfolio• Video Production http://tinyurl.com/7vbvag2
Section 3. Instructional Materials and Resources UtilizedThe instructor has carefully selected a varietyof materials and material formats to represent course content and enable students to meet relevant learning outcomes.
Objectives Example3.a Instructor provides students with adequate Instructor includes instruction in the syllabus or elsewhere in the course astime and notice to acquire course materials. to acquire course materials including textbooks, and other types of external resources.3.b Syllabus lists whether textbooks and Instructor separates the materials and labels them as either required ormaterials are required or recommended. recommended.3.c Instructor articulates the purpose of all For required and recommended materials, there are brief statements as tomaterials as to how they are related to the the value/purpose in meeting student learning objectives/outcome(s). Ifcourse and module learning objectives. external links/websites are used, the links should be self-evident or a short description of the specific link needs to be provided instead of posting a general link for students to explore.3.d When possible, instructor provides s Course materials include both the Open Educational Resources (e.g.options in terms of how students acquire MERLOT) and external materialscourse materials, including Open EducationalResources (e.g., MERLOT).3.e There is a variety of instructional material Materials types include PowerPoint, videos, text. Multiple perspectivestypes and perspectives, while not overly relying refer to different opinions from scholars in the field.on one content type such as text.3.f All resources and materials used in the These resources and materials include text, images, tables, videos, audio,course are appropriately cited. and website.
Section 4. Student Interaction and Community (Course Design)Addresses how the instructors provide opportunities for students to interact with the content, peers, instructors and the LMS and promote students to become active learners and build the online community.
Section 5. Facilitation and Instruction (Course Delivery)Instructor facilitates the course and communicates with students frequently and engages them to be active learners. Instructor actions reinforce the development of a sense of community among course participants.
Section 6.Technology for Teaching and Learning Instructor utilizes technology to effectively deliver course content, engage students in learningactivities (individual, student-to-student, instructor-to-student) and for students to express themselves or demonstrate learning.
Section 7.Learner Support and ResourcesLearner Support and Resources refers to program, academic, and/or technical resources available to learners.
SECTION 8 ACCESSIBILITY AND UNIVERSAL DESIGNThe course utilizes principles of accessibility anduniversal design that are critical to some learners, as well as offering benefits to all learners. We strongly recommend that instructors contact the Universal Design Center (or similar) for assistance and information about this section.
Students with DisabilitiesUpon identifying themselves to the instructor andthe university, students with disabilities willreceive reasonable accommodation for learningand evaluation. For more information, contactServices to Students with Disabilities in Building,Room (x4232).
Universal Design Statement“As your instructor, I feel I have a responsibility to doeverything within reason to actively support a wide rangeof learning styles and abilities. As such, I have appliedthe principles of Universal Design for Learning to thiscourse. Feel free to discuss your progress in the coursewith me at any time. In addition, if you require anaccommodation, submit your verified accommodationsform to me during the first two weeks of the course.”
Section 9 Course Summary and Wrap-up The course gives students an opportunity to summarize the semester, and establish the connection with other courses, and preparesstudents to start the next phase of their journey.
Questions Regarding Faculty Instrument and Process?
Gathering Student Ratings• Online evaluations due December 15, 2012• Minimal demographic information• Confidential (coordinator and instructor only)• Ratings submitted directly to QOLT Central• Process will be streamlined via Likert items o Optional open-ended at end of each section• Results used toward determining most exemplary per campus and across system.
Purpose of Student Ratings1. Identify any gaps between instructor’s perception and actual student experience.2. Make changes to courses based on evidence3. Use the data to receive administrative support to teach online or hybrid
Student ratings from the Quality Online Learning and Teaching (QOLT) survey Dora Preminger Dept. Physics & Astronomy • Li Wang Faculty Technology Center • California State University Northridge
The course Physics 220A – Classical Mechanics Calculus-‐based (Engineers) Why use the QOLT survey? • New hybrid format Is the course well-‐designed? (Survey makes QOLT rubric • Is it viable? concrete) Perceptions of students • Improve for future? Weak/missing elements?
Sample Results: Instructional Materials and Resources The instructor uses a variety of All materials are meaningful to me as instruc9onal material types and presents to how they are related to helping me mul9ple perspec9ves, while not overly achieve the course learning goals and relying on one content type such as text. objec9ves. Some Student comments: • There are way too many course materials for this course...more so than the non-‐hybrid counterpart which would already have been borderline too expensive. • The Ac9ve physics assignments are very confusing. I was not able to understand how to properly use it 9ll the instructor told me that should use the diﬀerent browser to use it properly.
Sample Results: Students Interaction and Community At the beginning of the course, GeBng I ﬁnd the naviga9on throughout the online to know other course par9cipants components of the course is reasonable, gave me a sense of belonging in the intui9ve and straighEorward. course. The learning ac9vi9es encourage me to interact with the instructor, my peers, and the course content frequently.
Sample Results: Overall Satisfaction Please rate your overall course Would you recommend this course learning experience. to your friend? Least………………….Most sa9sﬁed No……………………….Yes, deﬁnitely!
Summary Survey results: • Nice breakdown of the diﬀerent elements of the course – helps me see how I am ﬁBng them together (or not!) • Easy, quick view of statistics • Hear student voices • View of overall student experience • Pinpoint areas for course improvement
AFTER JANUARY 20• Each Campus Coordinator is given their data set within 1 week• They take 3 weeks to review the self-evaluations and respective student rating• With permission, may use sysadmin course access• They rank-order Top 5 and submit with their comments to QOLT Review Panel via survey• QOLT Review Panel begins with #1 from each campus as CSU “Top 15.” Reviews #2s next to see if any elevate. Followed by #3s, etc. Top 20 result.• Review panel "triangulates" between faculty, student, and coordinator data.
Recognition Process, After March 15• All faculty who submit a full self-evaluation receive a letter of participation from Campus Coordinator (draft provided to Campus Coordinator)• Top 5 designated from each campus receive an additional letter of recognition from CSU ATS.• Top 20 systemwide receive additional letter of recognition from CSU ATS and are featured at website and via QOLT dissemination (e.g., ITL newsletter; spring webinar; CATS)