• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Online Course Assessment Methods: How Effective Are They? (ppt)
 

Online Course Assessment Methods: How Effective Are They? (ppt)

on

  • 981 views

 

Statistics

Views

Total Views
981
Views on SlideShare
981
Embed Views
0

Actions

Likes
1
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Online Course Assessment Methods: How Effective Are They? (ppt) Online Course Assessment Methods: How Effective Are They? (ppt) Presentation Transcript

    • Online Course Assessment Methods: How Effective Are They? Dr. Kim Marie McGinnis
    • My Background
      • Doctorate in Educational Leadership and Policy Analysis, ETSU
      • Master’s in Education, WCU
      • BS in Landscape Horticulture, NCSU
      • Director of Occupational Extension Training, Carteret Community College
      • Dean of Technical and Vocational Programs, Mayland Community College
    • My Interest in this Study
      • I develop Internet courses
      • I want to continually improve the quality and effectiveness of the courses that I teach
      • I am also an Internet course student and have questions about the choices of assessments that have been used in those courses
      • Assessment is a very current accountability issue and distance education is becoming a common method of instruction
    • Need for this Study
      • Assessment has become an integral part of accountability in the American educational system
      • Distance education is the fastest growing form of education
      • With increased emphasis on accountability in general and increased scrutiny of online teaching and learning, issues of assessment have taken on more importance than ever before (Comeaux, 2005)
    • Research Questions
      • Academic discipline: Are there differences in assessment methods being used among faculty who teach in different academic disciplines in the online environment?
      • Learning Objectives being met: Are there differences in perceived effectiveness of the assessment methods being used among individual instructors in determining if the course learning objectives have been met?
    • Research Questions
      • Internet course development training: Are there differences in assessment methods used between those online instructors who received training in Internet course development as compared to those who did not?
      • Number of assessments per course: Are there differences in the number of different types of assessments being used per course by each instructor?
    • Research Questions
      • Years teaching Internet courses: Are there differences in the types of assessments being used by online instructors who have been teaching in the online environment for more than three years, as compared with instructors who have been teaching in the online environment for three or fewer years?
    • Research Questions
      • Number of Internet courses per year: Are there differences in the types of assessments being used by online instructors who teach more than one Internet course per year and instructors who teach only one Internet course per year?
    • Research Design
      • Primary data analysis was conducted based on responses to an original survey document developed by the researcher
      • The survey was conducted online and sent to all online instructors who had taught an Internet course or a web-enhanced course during the 2004-2005 academic year at the 15 Western North Carolina community colleges that serve the Appalachian region (the population - 371)
      • 174 responses were received (47%)
    • The Survey
      • The survey consisted of 16 questions
      • Questions 10 and 12 were open-ended questions
      • Question 1 identified the academic department for which the instructor taught
      • Questions 2,3,4, and 5 addressed the number of unduplicated Internet courses that the instructor taught during different academic years
      • Question 6 identified the assessment methods that individual instructors were using in the Internet courses they taught
    • The Survey
      • Question 8 required a yes or no response to receiving training in Internet course development
      • Question 9 focused on the area in which the training was received
      • Question 11 provided the number of different types of assessment being used by individual instructors per course
      • Questions 13-16 retrieved demographic information
    • Demographics
      • Gender: Of 167 responses, 71 respondents were male (40.8%) and 96 were female (55.2%)
      • Age: Of 164 responses, the mean age was 44.65 with a range from 25-76 years old
      • Academic degree: Of 166 responses, 8 had an associate’s degree (4.6%), 19 had a bachelor’s degree (10.9%), 125 had a master’s degree (including EdS) (71.8%), and 14 had a doctoral degree (8.0%)
    • Demographics
      • Years experience in education: Of 169 responses, the mean number of years of experience in education was 13.95 with a range from 1-45 years
    • Research Question 1
      • Are there differences in assessment methods being used among faculty who teach in different academic disciplines in the online environment?
      • Chi-square with a two-way contingency table analysis was conducted for each assessment method
    • Research Question 1
      • Academic discipline and portfolio use were not found to be significantly related
      • Uses portfolio: Business (21.3%), Vocational (23.1%), Health Occupations (0%), Arts (20%), Public Safety (0%), Continuing education (0%), Social Sciences (18.2%), Hard Sciences (0%), Other (0%)
    • Research Question 1
      • Academic discipline and true/false test use were found to be significantly related
      • Uses true/false tests: Business (68.0%), Vocational (76.9%), Health Occupations (57.1%), Arts (30%), Public Safety (60%), Continuing Education (100%), Social Sciences (27.3%), Hard Sciences (56.3%), and Other (100%)
    • Research Question 1
      • Academic discipline and multiple-choice test use were found to be significantly related
      • Uses multiple-choice tests: Business (92%), Vocational (92.3%), Health Occupations (100%), Arts (63.3%), Public Safety (100%), Continuing Education (100%), Social Sciences (81.8%), Hard Sciences (87.5%), and Other (100%)
    • Research Question 1
      • Academic discipline and short-answer test use were found to be significantly related
      • Uses short-answer tests: Business (44%), Vocational (84.6%), Health Occupations (57.1%), Arts (36.7%), Public Safety (60%), Continuing Education (100%), Social Sciences (31.8%), Hard Sciences (62.5%), Other (100%)
    • Research Question 1
      • Academic discipline and essay use were not found to be significantly related
      • Uses essay tests: Business (28%), Vocational (53.8%), Health Occupations (42.9%), Arts (56.7%), Public Safety (80%), Continuing Education (50%), Social Sciences (50%), Hard Sciences (37.5%), Other (100%)
    • Research Question 1
      • Academic discipline and discussion use were found to be significantly related
      • Uses discussion: Business (48%), Vocational (84.6%), Health Occupations (57.1%), Arts (80%), Public Safety (100%), Continuing Education (100%), Social Sciences (90.9%), Hard Sciences (37.5%), Other (100%)
    • Research Question 1
      • Academic discipline and individual project use were not found to be significantly related
      • Uses individual projects: Business (74.7%), Vocational (69.2%), Health Occupations (57.1%), Arts (73.3%), Public Safety (80%), Continuing Education (50%), Social Sciences (40.9%), Hard Sciences (6.3%), Other (0%)
    • Research Question 1
      • Academic discipline and group project use were found to be significantly related
      • Uses group projects: Business (24%), Vocational (30.8%), Health Occupations (71.4%), Arts (53.3%), Public Safety (80%), Continuing Education (0%), Social Sciences (40.9%), Hard Sciences (6.3%), Other (0%)
    • Research Question 1
      • Academic discipline and problem-solving activity use were not found to be significantly related
      • Uses problem-solving activities: Business (44%), Vocational (23.1%), Health Occupations (28.6%), Arts (40%), Public Safety (20%), Continuing Education (50%), Social Sciences (27.3%), Hard Sciences (43.8%), Other (0%)
    • Research Question 1
      • Academic discipline and self-assessment use were not found to be significantly related
      • Uses self-assessment: Business (14.7%), Vocational (15.4%), Health Occupations (28.6%), Arts (20%), Public Safety (20%), Continuing Education (0%), Social Sciences (4.5%), Hard Sciences (0%), Other (0%)
    • Research Question 1
      • Academic discipline and other assessment use were not found to be significantly related
      • Uses other assessments: Business (6.7%), Vocational (7.7%), Health Occupations (0%), Arts (23.3%), Public Safety (0%), Continuing Education (0%), Social Sciences (18.2%), Hard Sciences (12.5%), Other (0%)
    • Research Question 2
      • Are there differences in perceived effectiveness of the assessment methods being used among individual instructors in determining if the course learning objectives have been met?
      • A chi-square test of frequencies was used
      • There was a statistically significant difference in the perceptions of effectiveness of all of the assessment methods with the exception of self-assessment and the category of other
    • Research Question 2
      • Figure 1: Bar Graph of the Perceived Effectiveness of Portfolio Assessment (least effective N = 7, somewhat effective N = 5, effective N = 12, more effective N = 13, most effective N = 22)
    • Research Question 2
      • Figure 2: Bar Graph of the Perceived Effectiveness of True/False Tests (least effective N = 12, somewhat effective N = 19, effective N = 58, more effective N = 22, most effective N = 7)
    • Research Question 2
      • Figure 3: Bar Graph of the Perceived Effectiveness of Multiple-Choice Tests (least effective N = 6, somewhat effective N = 16, effective N = 61, more effective N = 49, most effective N = 23)
    • Research Question 2
      • Figure 4: Bar Graph of the Perceived Effectiveness of Short Answer Tests (least effective N = 2, somewhat effective N = 2, effective N = 28, more effective N = 49, most effective N = 23)
    • Research Question 2
      • Figure 5: Bar Graph of the Perceived Effectiveness of Essay Tests (least effective N = 0, somewhat effective N = 2, effective N = 10, more effective N = 41, most effective N = 41)
    • Research Question 2
      • Figure 6: Bar Graph of the Perceived Effectiveness of Discussion Questions (least effective N = 3, somewhat effective N = 5, effective N = 27, more effective N = 54, most effective N = 39)
    • Research Question 2
      • Figure 7: Bar Graph of the Perceived Effectiveness of Individual Projects (least effective N = 6, somewhat effective N = 0, effective N = 12, more effective N = 37, most effective N = 72)
    • Research Question 2
      • Figure 8: Bar Graph of the Perceived Effectiveness of Group Projects (least effective N = 4, somewhat effective N = 13, effective N = 15, more effective N = 29, most effective N = 15)
    • Research Question 2
      • Figure 9: Bar Graph of the Perceived Effectiveness of Problem- Solving Activities (least effective N = 4, somewhat effective N = 3, effective N = 8, more effective N = 30, most effective N = 43)
    • Research Question 2
      • Figure 10: Bar Graph of the Perceived Effectiveness of Self Assessment (least effective N = 11, somewhat effective N = 13, effective N = 16, more effective N = 10, most effective N = 3)
    • Research Question 2
      • Figure 11: Bar Graph of the Perceived Effectiveness of Other Assessment Methods (least effective N = 5, somewhat effective N = 4, effective N = 3, more effective N = 12, most effective N = 9)
    • Research Question 3
      • Are there differences in assessment methods used between those online instructors who received training in Internet course development and those that did not?
      • A chi-square two-way contingency table analysis was used to compare each assessment method individually with training
    • Research Question 3
      • Training and uses portfolio were not significantly related
      • 16.1% of instructors who received training used portfolio
      • 22% of instructors who did not receive training used portfolio
    • Research Question 3
      • Training and true/false test use were not found to be significantly related
      • 55.6% of instructors who received training used true/false tests
      • 52% of instructors who did not receive training used true/false tests
    • Research Question 3
      • Training and multiple-choice test use were not found to be significantly related
      • 87.9% of instructors who received training use multiple-choice tests
      • 78% of instructors who did not receive training use multiple-choice tests
    • Research Question 3
      • Training and short-answer test use were not found to be significantly related
      • 47.6% of instructors who received training use short-answer tests
      • 50% of instructors who did not receive training use short-answer tests
    • Research Question 3
      • Training and essay test use were not found to be significantly related
      • 41.9% of instructors who received training use essay tests
      • 40% of instructors who did not receive training use essay tests
    • Research Question 3
      • Training and discussion use were found to be significantly related
      • 70.2% of instructors who received training use discussion
      • 48% of instructors who did not receive training use discussion
    • Research Question 3
      • Training and individual project use were not found to be significantly related
      • 71.8% of instructors who received training use individual projects
      • 70% of instructors who did not receive training use individual projects
    • Research Question 3
      • Training and group project use were not found to be significantly related
      • 36.3% of instructors who received training use group projects
      • 24% of instructors who did not receive training use group projects
    • Research Question 3
      • Training and problem-solving activity use were not found to be significantly related
      • 41.9% of instructors who received training use problem-solving activities
      • 30% of instructors who did not receive training use problem-solving activities
    • Research Question 3
      • Training and self-assessment use were not found to be significantly related
      • 16.1% of instructors who received training use self-assessment
      • 6% of instructors who did not receive training use self-assessment
    • Research Question 3
      • Training and other methods of assessment use were not found to be significantly related
      • 12.1% of instructors who received training use other methods of assessment
      • 8% of instructors who did not receive training use other methods of assessment
    • Research Question 4
      • Are there differences in the number of different types of assessments being used per course by each instructor?
      • A chi-square frequencies test was conducted
    • Research Question 4
      • There was a significant difference in the number of different types of assessments being used per course by each instructor
      • The number of different types of assessments being used per course per instructor ranged from 1-20 with a mean of 3.6
    • Research Question 5
      • Are there differences in the types of assessments being used by online instructors who have been teaching in the online environment for more than three years as compared with instructors who have been teaching in the online environment for three or fewer years?
    • Research Question 5
      • A chi-square two-way contingency table analysis was conducted comparing each individual assessment method with the length of time teaching online courses
    • Research Question 5
      • Length of time teaching online courses and portfolio use were not found to be significantly related
      • 17.8% of online instructors who have taught more than 3 years use portfolio
      • 18.2% of online instructors who have taught 3 or fewer years use portfolio
    • Research Question 5
      • Length of time teaching online courses and true/false test use were not found to be significantly related
      • 54.6% of online instructors who have taught for more than 3 years use true/false tests
      • 55.9% of online instructors who have taught for 3 or fewer years use true/false tests
    • Research Question 5
      • Length of time teaching and multiple-choice test use were found to be significantly related
      • 85.1 % of online instructors who have taught for more than 3 years use multiple-choice tests
      • 85.3% of online instructors who have taught for 3 or fewer years use multiple-choice tests
    • Research Question 5
      • Length of time teaching and short-answer test use were not found to be significantly related
      • 48.3% of online instructors who have taught for more than 3 years use short-answer tests
      • 48.8% of online instructors who have taught for 3 or fewer years use short-answer tests
    • Research Question 5
      • Length of time teaching and essay test use were not found to be significantly related
      • 41.4% of online instructors who have taught for more than 3 years uses essay tests
      • 41.2% of online instructors who have taught for 3 or fewer years uses essay tests
    • Research Question 5
      • Length of time teaching and discussion use were not found to be significantly related
      • 63.8% of online instructors who have taught for more than 3 years use discussion
      • 64.1% of online instructors who have taught for 3 or fewer years use discussion
    • Research Question 5
      • Length of time teaching and individual project use were not found to be significantly related
      • 71.3% of online instructors who have taught for more than 3 years use individual projects
      • 71.2% of online instructors who have taught for 3 or fewer years use individual projects
    • Research Question 5
      • Length of time teaching and group project use were not found to be significantly related
      • 32.8% of online instructors who have taught for more than 3 years use group projects
      • 32.9% of online instructors who have taught for 3 or fewer years use group projects
    • Research Question 5
      • Length of time teaching and problem-solving activity use were not found to be significantly related
      • 38.5% of online instructors who have taught for more than 3 years use problem-solving activities
      • 38.8% of online instructors who have taught for 3 or fewer years use problem-solving activities
    • Research Question 5
      • Length of time teaching and self-assessment use were not found to be significantly related
      • 13.2% of online instructors who have taught more than 3 years use self-assessment
      • 13.5% of online instructors who have taught 3 or fewer years use self-assessment
    • Research Question 5
      • Length of time teaching and other assessment method use were not found to be significantly related
      • 10.9% of online instructors who have taught more than 3 years use other assessment methods
      • 10.6% of online instructors who have taught 3 or fewer years use other assessment methods
    • Research Question 6
      • Are there differences in the types of assessments being used by online instructors who teach more than one Internet course per year as compared with instructors who teach only one Internet course per year?
      • A chi-square two-way contingency table analysis was used
    • Research Question 6
      • The number of online courses taught per year was not significantly related to portfolio, true/false tests, short answer tests, essay, discussion, individual projects, problem-solving activities, self-assessment or other assessment methods
    • Research Question 6
      • The number of online courses being taught per year was significantly related to multiple-choice tests and group projects
      • 73.2% of online instructors who teach only 1 Internet course use multiple-choice tests
      • 85.1% of online instructors who teach more than 1 Internet course per year use multiple-choice tests
    • Research Question 6
      • 23.2% of online instructors who teach only 1 Internet course per year use group projects
      • 32.8% of online instructors who teach more than 1 Internet course per year use group projects
    • Qualitative Data
      • Please describe and indicate how effective the training you received was?
      • 67% stated that the training was effective
      • 18% stated that the training was ineffective
    • Qualitative Data
      • Themes:
        • Those trained would have liked to have training in assessment and pedagogy in addition to BlackBoard/technology
        • Training was technology based only
        • Training was minimal
        • A distance education coordinator that was regularly available for follow up training was extremely helpful
    • Qualitative Data
      • If you have taught Internet courses for more than one year, have your methods of assessment changed over time? Why?
      • 61% responded that their methods have changed over time
      • 38% responded that their methods have not changed over time
    • Qualitative Data
      • Themes:
        • A variety of assessment methods need to be used in each course to ensure that the course objectives are being met
        • Alternative assessment methods are more reliable indicators of course effectiveness
        • Lack of time – some instructors are using less effective objective testing because of a lack of time
    • Qualitative Data
      • “Previously, I avoided objective (multiple-choice, true/false) assessments and used only short answer/discussion-type questions; however, they became too time consuming to grade considering the number of students in my sections (often 30).”
    • Conclusions
      • There is a difference between some academic disciplines in relation to the type of assessment methods being used in online courses
      • There is a difference in perceived effectiveness of assessment methods among the individual instructors surveyed
    • Conclusions
      • The survey results confirmed that objective testing is not considered the most effective method of assessment but several instructors still use this method because of time constraints
      • The survey results confirm that a variety of assessment methods need to be used within each Internet course to determine the effectiveness of the course
    • Conclusions
      • There was no difference in the assessment methods being used by those instructors who received training and those who did not
      • There is a difference in the number of types of assessment being used by online instructors
    • Conclusions
      • The survey data indicated that there was not a difference in assessment methods being used by instructors who have taught for more than 3 years as compared to instructors who have taught for 3 or fewer years
      • There is a difference in some of the types of assessments being used by instructors who teach more than 1 Internet course per year
    • Implications for Practice
      • The findings of this study indicated that online instructors should be trained in assessment methods and pedagogy in addition to the course management system and technology
      • A policy needs to be developed within the NCCC system that sets guidelines for workload for Internet course instructors so that the most effective methods of assessment will be used in the online environment
    • Implications for Practice
      • A policy needs to be developed within the NCCC system that places caps on the number of students who are allowed to be in one course section of an online course so that the instructor has time to use the more effective means of assessment in online courses
    • Implications for Practice
      • The findings of the study indicate that a distance education coordinator should be hired at all institutions and this individual should provide monthly training and individual consultation to online instructors to ensure that the instructors are adequately trained and comfortable in the online environment
    • Recommendations for Future Research
      • Similar studies should be conducted in other parts of the state of North Carolina to confirm results from the western part of the state
      • This study should be replicated after the groups of respondents have received training in assessment methods
    • Recommendations for Future Research
      • This study should be conducted by surveying students instead of instructors to see if the students’ perspectives match the instructors’ perspectives
      • A qualitative study investigating perceptions of effectiveness of assessment methods in the online environment should be undertaken
    • Recommendations for Future Research
      • Similar studies should be conducted in other states