Download to read offline
Assessment of online learning
- 1. Glori Hinck, RD,MS, DC Assistant Professor, Methods Department Northwestern Health Sciences University Bloomington, MN USA
- 2. Convenience Material readilyavailable Active learning Decrease class room “seat” time
- 3.
- 4.
- 5.
- 6.
- 7. Large classes (60to 130) Solution: Automatically graded assessments Utilize TA’s and co-instructors for grading Limit use of discussion forums Use group mode when feasible
- 8. Can online course delivery be as effective a method of teaching in the chiropractic setting as a course based on a traditional , lecture-format delivery?
- 9. “ No SignificantDifference Phenomenon” Studies comparing results of various teaching methods have found “no significant difference” in student outcomes when using various types of technology and education delivery systems Controversy exists
- 10. Bassano (2005) Nosignificant difference in learning occurred between students with access to an online self-guided radiology tutorial and those without this access Goubran and Vijamury (2007) The use of an online interactive atlas of histology led to significantly higher average laboratory examination scores Journal of Chiropractic Education
- 11. Mean final exam scores for students taking the online course are statistically equivalent to, or greater than, the mean final exam scores of students taking the traditional course in the previous six trimesters .
- 12. “ Same” basiccore content “ Same” examination Compared mean exam scores
- 13. What was “wrong”with this study design?
- 14. New instructor Newdelivery method Instructor, Students, Campus Technology issues Exam does not reflect variations in content Exam score weighting 33% vs. 50%
- 15. Physical Diagnosis course moved from Term 3 to Term 4 during the Fall ‘07 term Instructors noticed impact on clinical reasoning in Methods 3 (Term 3) Hypothesized indirect effect on exam scores
- 16.
- 17. Traditional (Sixtrimesters) Mean Final Exam Score: 73.4% Online (One trimester) Mean Final Exam Score: 72.5% Analysis indicates statistical equivalence between groups: mean final exam scores ( t= 1.09 p=0.89 )
- 18. Online (Onetrimester) Mean Midterm Exam Score: 75.1% Traditional (Six trimesters) Mean Midterm Exam Score: 79.2% Analysis indicates traditional midterm score statistically greater than online midterm score ( t= 4.69 p=0.00 )
- 19. F’07: Last traditionalMidterm: 76.1% Final: 71.5% S’08: First online Midterm: 75.1% Final: 72.5% Analysis indicates statistical equivalence between groups: Midterm: ( t= 0.77 p=0.44 ) Final: ( t= -0.77 p=0.44 ) Neither group had PDX concurrently
- 20. Traditional (PDX) (5trimesters) Midterm 79.7% Final: 73.7% Traditional (No PDX) (2 trimesters) Midterm 76.1% Final: 71.5% Analysis indicates concurrent PDX is associated with statistically greater midterm and final exam scores than no-PDX Midterm: (t=3.51 p=0.00) Final: (t=2.33 p=0.02)
- 21. No significant difference:All Traditional vs. Online final exam F’07 Traditional vs. Summer ‘08 Online midterm and final Traditional significantly better: All Traditional vs. Summer ‘08 Online midterm Concurrent PDX scores significantly greater
- 22. The evidence supportsour alternative hypothesis that online course delivery can be an effective method of teaching in the chiropractic setting
- 23. What aspects ofonline delivery are most successful? Item analysis Any changes needed in specific areas? Student satisfaction surveys and focus groups
- 24.
- 25. Mean midterm exam scores for students taking the online course are statistically equivalent to, or greater than, the mean midterm exam scores of the students taking the traditional course in the previous six trimesters .
- 26. Mean midterm and final exam scores for students taking the online course are statistically equivalent to, or greater than, the mean midterm and final exam scores of the students taking the traditional course in the immediately-previous trimester
- 27. Mean midtermand final scores for students taking the traditional course with a concurrent physical diagnosis course are statistically equivalent to, or greater than, the mean midterm and final exam scores of students who took the traditional course with no concurrent physical diagnosis course. (Assessment of confounding factor in traditional-online comparisons)
Editor's Notes
- #2 Good afternoon. During the summer of 2008 NWHSU presented their first primarily online course. I am here today to talk about a study that looked at learning outcomes with this method of course delivery. Why did we do this study? Before we increased our online offerings at Northwestern we wanted some assurance that it was feasible, and that equivalent learning could occur.
- #3 Why are we here today? Why bother changing our format to teach online? Allows learning to take place at locations and times convenient to students Material available for students to review if they miss lecture or don’t understand content Supports use of active learning Decrease class room “seat” time
- #4 During the summer of 2008 Methods 3: Cervical and Thoracic Manual Therapies was presented in a hybrid, asynchronous format. While both lecture and lab material were presented online, the laboratory sessions continued to be taught in the traditional format while the online material replaced the bulk of the actual classroom lectures. Three in-class sessions were required- the introduction to the course, the midterm, and a group case study. In the future we anticipate the entire lecture component presented online with the exception of proctored in-class midterm and final examinations.
- #5 The course was categorized as hybrid for two reasons- Lecture attendance was required 3 times and you any technique instructors will be happy to know that laboratory sessions were held exactly the same as previously. The only difference with the labs was that the students were given access to additional materials via Moodle.
- #6 This slide shows a typical weekly module as it is presented on Moodle, our course management system. Each week is divided into required online material to view; required reading from books or journal articles; The various assessments that the student will be responsible for that week- both online and in the lab; and the Lab scheduled material for that week. Online lecture content was primarily presented in the format of narrated PowerPoint slides with readings from the textbook, course notes, and journal articles supplementing this material. Although not a part of this module, Online case studies in a variety of formats were an important part of the course.
- #8 Large class sizes are a challenge in the implementation of online courses. Typically online courses are limited to 20-30 students or fewer. Northwestern’s class sizes vary from 80 to 120 students. This course was specifically designed to overcome this obstacle. Many of the assessments are automatically graded by the course management system. Others are designed so that others can aid in grading.
- #9 With this research we wanted to answer the question: Can online course delivery be as effective a method of teaching in the chiropractic setting as a course based on a traditional , lecture-format delivery?
- #10 A review of the literature suggests that it can. A review of the literature finds extensive reference to the “No significant Difference Phenomenon” which basically states that studies comparing results of various teaching methods have found no significant differences in student outcomes when using various types of education delivery systems. However, not every study finds positive results. There is some controversy with this concept. There are so many variables to control- same instructors, same material, same assessments. Are you using the right assessment.
- #11 No studies specifically comparing learning outcomes between an online course presented via a course management system and a traditional classroom format in the chiropractic setting were found. However, two studies regarding online learning were recently published in the Journal of Chiropractic Education. Bassano found that no significant difference in learning occurred between students with access to an online self-guided radiology tutorial and those without this access. Goubran and Vijamury found that the use of an online interactive atlas of histology led to significantly higher average laboratory examination scores.
- #12 To answer our research question, we formulated a number of hypotheses. The primary research hypothesis was: We also formulated several secondary hypotheses also looking at the midterm scores and different cohorts of students.
- #13 In a nutshell, we took the same basic course course content with minor adaptations as necessary to convert it to an online format and used essentially the same examination that had been used previously with no changes in content. We then compared average exam scores between various groups of students.
- #15 We believe that a number of factors potentially biased against online learning. The instructor was new to both the content and to online delivery. The examination that was used did not reflect her style and suble variations in content. The instructor was new to the content as well as to online course delivery and this was the first online course in the history of NWHSU. Although the content was similar to the previous lecture format, there were still significant differences in presentation and delivery. The midterm examination did not reflect these differences as the the previous lecture examination was used in order to facilitate this study. An additional reason for decreased performance is hypothesized. With the online format 200 points are added to the course in the form of active learning assignments. These points decrease the weighting of the exam scores from 50% of the overall course grade to 33%. When the exam has less weight, students have a tendency to direct less time toward this assessment and towards higher stakes examinations in their other courses.
- #16 Just to make it interesting, there was one additional confounding variable…
- #17 So, what did we learn?
- #18 Analysis comparing mean final exam scores between 6 trimesters of traditional students and one trimester of online students found no statistical difference- proving our primary hypothesis.
- #19 We did find a significant difference in midterm exam scores between the traditional and online students with the traditional students scoring significantly higher at 79.2% vs. 75.1% for the online students.
- #20 When we looked at the last time that Methods 3 was taught in the traditional format and compared it to the first time that the course was taught online we found no significant difference between groups for midterm or final exam scores. The significance of this comparison was that neither of these groups had PDX concurrently.
- #21 Due to the hypothesized effect of the PDX curriculum change, we took our analysis a step further and specifically compared terms with concurrent PDX to those who did not have PDX and found significantly greater exam scores in the terms with concurrent PDX.
- #22 What is the bottom line? Even with the potential biases against online learning inherent in this study design, our results still suggest that “no significant difference” exists between online delivery and the traditional lecture format for this course. In fact, the effects of a curriculum change are found to potentially be more significant than changes in delivery method. Online delivery learning curve/Technology Issues ? PDX effect? Factors biasing against online?
- #23 Now what?
- #24 Rather than comparing apples and oranges, i.e. comparing online to traditional. Let’s look at ….
- #26 The secondary research hypothesis addressed the mid term exam. Greater emphasis was placed on the final exam due to the initial learning curve for both the instructor and the students and greater technology issues during the first half of the term.