Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ITC12 Pre-conference Workshop: Evaluating Online Faculty2-slideshare


Published on

Slides by Barry Dahl, Excellence in e-Education for ITC eLearning 2012 Conference in Long Beach

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

ITC12 Pre-conference Workshop: Evaluating Online Faculty2-slideshare

  1. 1. “Quality”in Online Education
  2. 2. Barry Dahl dot com
  3. 3. Quality Definition?high grade; superiority; excellence
  4. 4. Qualityin Manufacturing Definition?What does ISO say? InternationalOrganization for Standardization
  5. 5. ISO 8402:1986• This ISO standard defines quality as “the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs.” Does that actually mean anything?
  6. 6. Another Definition• In manufacturing, a measure of excellence or a state of being free from defects, deficiencies, and significant variations, brought about by the strict and consistent adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements. Just like education, right?
  7. 7. Quality in Education Definition?High standards, consistently applied,with efficiency or “value for money”
  8. 8. Quality in Education Definition?Or, what D. Green said in 1994What is quality in higher education?Buckingham, U.K.: SRHE and The Open University Press.
  9. 9. What is quality in higher education?• …quality is a relative concept, ....different interest groups or ‘stakeholders’ in higher education have different priorities and their focus of attention may be different. The best that can be achieved is to define as clearly as possible the criteria that each stakeholder uses when judging quality, and for these competing views to be taken into account when assessments of quality are undertaken (p. 17).
  10. 10. Qualityin Online Ed Definition?
  11. 11. E-Learning Quality = Quality Matters™ Right?
  12. 12. Through the use of rubrics and standards related to the quality of online courses (i.e. Quality Matters™), we are sufficiently addressing the questions about e-learning quality60% 1.Real ity40% 2.Myth
  13. 13. Quality Matters is Sufficient Learning• Um, no, it isn’t!!• Quality Matters looks at the quality of course design. Quality• That’s good, but it’s Concerns only one leg holding up the stool. Teaching Design
  14. 14. 3 Major Components of e-Quality Learning Level Is High Learning Assessment Teaching Level Course Design Is High Meets Standards
  15. 15. Possible Reasons? Learning Level Is Low Learning AssessmentTeaching Level Course Design Is High Meets Standards
  16. 16. Is This a Problem? Learning Level Is High Learning AssessmentTeaching Level Course Design Is High Below Standard
  17. 17. Independent Study, Perhaps? Learning Level Is High Learning Assessment Teaching Level Course Design Is Low Meets Standards
  18. 18. Beautiful, Just Beautiful Learning Level Is Low Learning AssessmentTeaching Level Course Design Is Low Meets Standards
  19. 19. Maybe, but unlikely Learning Level Is Low Learning AssessmentTeaching Level Course Design Is High Below Standard
  20. 20. Triple Ick – Just Start Over? Learning Level Is Low Learning Assessment Teaching Level Course Design Is Low Below Standard
  21. 21. QualityWe know it when we see it
  22. 22. Expectations about Completion
  23. 23. Weekly News Articles
  24. 24. CCRC Reports: Washington & Virginia
  25. 25. CCRC Reports: Washington & VirginiaWashington State community and technical colleges in the fall of 2004.Students were tracked for nearly five years, until the spring of 2009 “students were more likely to fail or withdraw from online courses than from face-to-face courses”“were slightly but significantly less likely to attain aneducational award or transfer to a four-year institution”
  26. 26. Say What?• “slightly but significantly less likely”• Translation … “I’m really, really sure that there’s a very small difference.”
  27. 27. CCRC Makes Big Headlines
  28. 28. CCRC Makes Big Headlines
  29. 29. What are ReasonableExpectations for Completion Rates for Online Learners? And how do you know? Is completion a measure of quality?
  30. 30. Online Student Satisfaction Is satisfaction a measure of quality?
  31. 31. Option 1 - PSOL
  32. 32. Why PSOL?
  33. 33. PSOL BasicsThere are 72 questions that comprise the PSOL.Average completion time is 15 minutes.26 Priorities statements (can add 10)7 Information sources about school/program11 Factors to enroll in the program3 Overall satisfaction questions14 Demographics questions (can add 1)NOTE: questions are answered on a 7-point Likert scale, where 7 is high. 34
  34. 34. The Importance of Importance
  35. 35. That’s all good, but…….
  36. 36. Are online students, as agroup, more satisfied than the on-campus students?
  37. 37. NOTE: SSI is the Noel-Levitz Survey for on-ground learners, PSOL is the Noel- 38Levitz survey for online learners.
  38. 38. NOTE: SSI is the Noel-Levitz Survey for on-ground learners, PSOL is the Noel- 39Levitz survey for online learners.
  39. 39. What drivestheir level ofsatisfaction?
  40. 40. How Well Do You Know Your Students? What do They Expect from Online Learning?
  41. 41. 2009 PSOL – Summary StatementSo far, how has your college experience met your expectations? 7=Much better than expected Overall score6=Quite a bit better than I expected 5.0 5=Better than I expected 4=About what I expected 3=Worse than I expected2=Quite a bit worse than I expected 1=Much worse than expected 0% 5% 10% 15% 20% 25% 30%
  42. 42. But seriously…
  43. 43. You Already Have Expectations• The questions are … – How clearly have they been communicated? – How easy are they for the following to find? •Students? •Faculty? <= We’ll focus here •Staff and Administration?
  44. 44. A Common Scenario• “What do you mean you expect interaction in online classes?”• “I’ve never heard that before.” This came from a 10-year veteran of online teaching at the college.
  45. 45. Course Design Rubrics Can HelpV. LEARNER INTERACTIONGeneral Review Standard: The effective design of instructor-learner interaction andmeaningful learner cooperation is essential to learner motivation, intellectual commitment,and personal development. (From LSC Course Design Rubric)Specific Review Standards PointsV.1 The course design provides learning activities tofoster instructor-student, content-student, and if 3appropriate, student-student interaction.V.2 The student requirements for course interactionare clearly articulated. 3V.3 Clear standards are set for instructor responseand availability (turn-around time for email, grades 2posted, etc.)
  46. 46. Course Design Rubrics Can Help• But they only go so far.• Did the interactions actually occur?• Did the feedback actually happen in a timely manner?• The design rubric is only part of the answer.
  47. 47. Do You Evaluate Teaching Quality?• If not, why not?• If so, what do you base the evaluation on?• How clear are the expectations of the factors upon which they will be evaluated?
  48. 48. Expectations for (of) Faculty Links to these pages at:
  49. 49. St. Petersburg College• Examples of posted expectations for faculty:• Respond to email within 24 hours, 5 of the 7 days a week at the instructor’s discretion.• Exam marked within 7 days of due date.• Assignment marked within 7 days of due date• Routine updates to all students every 7 days.
  50. 50. Lawrence Tech University• Examples of posted expectations for faculty:• Check the discussion forum daily and be sure to post responses to student contributions at least four times weekly.
  51. 51. Penn State• Examples of posted expectations for faculty:• The instructor is asked to grade and submit to students all digitally formatted assignments and exams within two business days of receipt.
  52. 52. Colorado CC Online• Examples of posted expectations for faculty:• As mandated by our NCA accreditation, CCCOnline courses cannot be self-paced. Therefore, CCCOnline requires faculty to create a Course Schedule that provides unit, week, and specific activity dates.• Instructor responds individually to all student introductory posts.
  53. 53. Lake Superior• Examples of posted expectations for faculty:• Course Outlines: There is only one official course outline for each course title offered at LSC. This means the following: – Delivery method is not generally considered during the curriculum approval process. – Each class syllabus should include the official course outcomes regardless of the delivery method employed. – Each class syllabus should include the official course description regardless of the delivery method employed.
  54. 54. College expectations of online faculty • Example - Final Exams:• Sample of possible topics: – Online faculty are expected to give final – Course design exams (or due dates for other types of final – Interaction w/students projects) during the final – Online office hours exam period as established and posted by – Feedback/response time the college. – Final exams – The final exam period typically consists of four – Proctored exams weekdays and may or may – Due dates/times not include a weekend. This schedule is posted on – Sick/Personal leave the college website and otherwise makes the information available to all faculty.
  55. 55. ComprehensiveFaculty Evaluations
  56. 56. Evaluation Components1. Self Evaluation2. Professional Development Plan3. Course Evaluations (by students)4. Course Observation (by admin)
  57. 57. Course Evaluations
  58. 58. Questions without Answers• Optional or required?• Formative or Summative?• Anonymous?• Incentives?• Short or long? (Brief or comprehensive?)• Comparable to on-campus evals?
  59. 59. Optional or Required?• Better practice: Required to opt-out if not interested/willing.• Through conditional release – force students to view the evaluation form – with ability to opt-out if desired.• Good practice: Purely optional (expect low response rates)
  60. 60. Formative or Summative?• Formative eval – mid-term assessment – Students provide feedback to instructor• Summative eval – end-of-term – Students provide feedback to faculty & admin• Better practice: Use both types
  61. 61. Anonymous?• Difficult for formative evals to be anonymous.• Important for students to believe that summative evals are anonymous – but many still won’t believe it.• Good practice: Do not use the course shell to give the eval if it is intended to be anonymous.
  62. 62. Incentives?• Good practice: impress upon students that their participation is an important part of the quality improvement process• Major issue: any kind of tangible incentive eliminates the possibility of true anonymity, or even the appearance thereof.
  63. 63. Short or Long?• An evaluation form that is longer than 10-15 questions will usually result in low response rates.• Good practice: 10 important questions.• If students are allowed to opt-out, they are more likely to opt-in if they see that the survey is very short and their time invested will be very small.
  64. 64. Comparable to On-Campus Evals?• How is this valuable?• What is your purpose for asking the same questions?• Good practice: gather data that are valuable to the online teaching and learning experience – whether or not they are comparable to on-campus data.
  65. 65. A GPA-type Rating Scale• 4 – Strongly agree• 3 – Agree• 2 – Neutral• 1 – Disagree• 0 – Strongly disagree• N/A – Choose not to answer, Don’t know, or Not applicable
  66. 66. Sample Statements• The instructor has created a course layout that is easy to navigate.• This course challenged me intellectually.• The instructor provided feedback in a timely manner.• The feedback received on my coursework was helpful.
  67. 67. More Sample Statements• There was agreement between the posted course objectives (in syllabus) and what was taught in the course.• Policies for determining grades in this course were clearly explained.• Overall, this instructor has created a valuable learning experience for students.
  68. 68. Eval Statements Tied to Rubric?• Instructor presence?• Expectations for interaction?• Effective use of media?• Positive learning environment?• Clear written communication?
  69. 69. Eval Statements Tied to Other?• PSOL Data? –Student assignments are clearly defined in the syllabus –The frequency of student and instructor interactions is adequate. –Faculty are responsive to student needs.• NSSE or CCSSE?
  70. 70. Classroom Observation An Administrator’s Right?
  71. 71. Basic Expectations – 1A
  72. 72. Basic Expectations – 1A
  73. 73. Basic Expectations – 1A
  74. 74. Basic Expectations – 1C
  75. 75. Basic Expectations – 1D
  76. 76. Basic Expectations – 1D
  77. 77. Learning Experiences – 2A
  78. 78. Learning Experiences – 2B
  79. 79. Activity & Interactivity – 3E
  80. 80. Evaluations, Assessments – 4B
  81. 81. Where is the Value in eLearning?• Why is XXCC engaged in eLearning?• What are the benefits to the school?• What are the driving factors?
  82. 82. To Make a Difference
  83. 83. “Quality”in Online Education Discussion Time