Assessment in the Basic Speech Course Purpose, Plan, Process,Continuous Improvement and Benefits
Workshop PresentersMyra Walters, ModeratorChair, Department of speech Communication and Foreign LanguagesDr. Katie Paschall, PresenterProfessor of Speech Communication, Collier County CampusDr. Kevin Coughlin, PresenterDirector of Research, Planning and Development
About Edison State College• Located in Southwest Florida• Has three campuses and one center• Provides instructional and support services for more than 25,000 credit and 3000 non- credit students
Oral Communication Assessment: Purpose• In May 2009, our basic speech courses were designated as part of the gen ed core• InSeptember, 2009 we were told we needed to assess the oral communication competency.
Developing an Assessment Plan• How will we record speeches?• How many speeches do we need to record?• How much will it cost to fund this assessment project?• Where will the money come from to support this assessment project?
Developing an Assessment Plan• What type of speech should we record?• What should be the required length of the assessment speech?• How will we record and prepare speeches for viewing?
The Assessment Plan• Held assessment workshop facilitated by Dr. John Fredericks from Miami Dade• Decided upon topic, type of speech and length of speech• Received assistance from Dr. Coughlin to determine the number of speeches to be recorded and which classes would be recorded.
The Assessment Plan• Faculty were sent e-mails notifying them that their classes were selected• Speeches were recorded using digital cameras• Edison online student assistants uploaded speech videos to the colleges learning management system into three separate playlistsSpeech Link
The Assessment Plan• Assessment teams were given one week to evaluate speeches• Rubrics with rating scores were delivered to the department chairs office• Nine sets of rubrics were delivered to the assessment office for interpretation and evaluation• Full time faculty met to discuss assessment results
Assessment Process: Overview•Pilot Study: focus on inter-rater correlations and reliability estimates (fully-crossed; small number of speeches)•Review: Speech faculty consider results and refine rubric and other aspects of rating process•Full Study: Focus on student performance; secondary focus on inter-rater correlations and reliability (nested or sort-of-crossed; all speeches)
Assessment Process: Sample• Population: During the Fall 2011 term, 57 sections of SPC 1017 and SPC 2023 were offered on 3 campuses and 2 centers• Sample considerations: We wanted a representative sample; we wanted as many speeches that the rating teams could feasibly assess• Sample: 5 sections of SPC 1017; 2 sections of SPC 2023; all students from each selected section; 167 speeches (835 minutes to rate)
Assessment Process: Pilot• Design Characteristics • 14 Speeches • Fully crossed – All raters rated all speeches • 9 raters (3 groups of 3) • 6 rubric dimensions or outcomes• Types of Results • 14 tables and a summary outline • Means and standard deviations (percentage scores?) • Inter-rater reliability estimates by rubric demension • 9 raters • 6 rubric dimensions or outcomes
Assessment Process: Pilot• Results • Consider the tables (that Pdf file entitled pilot . . .) • All raters (aggregated) • All Cronbach’s alphas > .80 • Highest = Nonverbal-Physical • Lowest = Language • Reliability for each sub-group was generally lower than the the entire group of raters • Pairwise correlations highlighted areas of disagreement as tot the meaning of each rubric dimension • Again . . . the tables
Assessment Process: Full Study• Design characteristics • Two foci: Student performance & rubric performance • Nested (somewhat crossed) • 127 speeches (93 after exclusions) • 9 raters nested in 3 groups (3 raters per group) • Each group rated between 40 and 55 speeches • 8 dimensions (up from 6); 4 levels of rating• Data considerations o For student performance, we included only speeches that received a rating from each rater from the relevant group o For reliability, we considered all observations
Assessment Process: Full Study• Types of results • 25 tables and a summary outline • 19 tables associated with student performance • Achievement of student learning outcmes • Differential performance across campuses/centers • Differential performance across dimensions • ANOVA’s, means, frequency distributions • 6 tables associated with inter-rater correlations and reliability • Giant correlation table • Individual reliability tables
Assessment Process: Full Study• Results Highlights • Frequency distributions indicate that students did their best in terms of Content • ANOVA yielded evidence that student performance differed significantly by campus in only 2 of 8 dimensions (outcomes): Body and Conclusion • Giant correlation table (show them the table) • Second rating group had the highest and lowest inter-rater correlations • Rater 3 and Rater 6 had the strongest levels of agreement • Cronbach’s alphas exceed .9 in all cases)
Continuous Improvement• Revised rubric to better assess the public speaking competencies• Revised directions for the final speech• Planned professional development through our TLC• Planning a workshop to increase rubric reliability• Developing a repository of sample student speeches for instructional purposes• Planning to repeat the assessment in Spring 2013
Benefits• It can be done!• We can do it!• Camaraderie among faculty• Meaningful discussions• Gained insight into what is being taught in the classroom• Development of relationships with other departments (e-learning, technical support, assessment office)