Capstone Assessment Project


Published on

This presentation examines inter-rater reliability of the HSP portfolio rubric in addition to student and faculty experiences with the Capstone course.

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Capstone Assessment Project

  1. 1. Capstone Portfolio Assessment Project Susan Kincaid and Jackie Baker-Sennett Department of Human Services and Rehabilitation Western Washington University This project was made possible through support from the Woodring College of Education Assessment Committee, WWU
  2. 2. Purpose of the Capstone Assessment Project (CAP) <ul><ul><ul><li>Review recent literature on inter-rater reliability in portfolio assessment practices </li></ul></ul></ul><ul><ul><ul><li>Examine the rater reliability of the HS capstone portfolio rubric within sections of the course and across program locations (Bellingham, Everett, Cyber) </li></ul></ul></ul><ul><ul><ul><li>Determine student perceptions of portfolio learning and assessment </li></ul></ul></ul><ul><ul><ul><li>Examine instructor’s experiences with the portfolio assessment process and thoughts about teaching the Capstone Portfolio course </li></ul></ul></ul>
  3. 3. Background: The HSP Curriculum As part of a comprehensive curriculum revision in 2006 the Human Services Program moved to a standards based curriculum, with student learning assessed through a comprehensive Capstone Portfolio.
  4. 4. HSP Capstone Portfolio <ul><ul><li>A culminating course, project, and benchmark assessment in the Human Services major </li></ul></ul><ul><ul><li>Assesses students’ knowledge and skills as they relate to CSHSE National Standards (Council for Standards in Human Services Education) and program outcomes in the areas of technology, information literacy, critical thinking, and writing proficiency. </li></ul></ul>
  5. 5. Portfolio Coursework <ul><li>All HSP students take a sequence of two courses: </li></ul><ul><ul><li>HSP 304: Introduction to Portfolio Learning (1 credit during first 5 credits in major) </li></ul></ul><ul><ul><li>HSP 495: Capstone Portfolio (4 credits during last 5 credits in major) </li></ul></ul>
  6. 6. While most Human Services programs in the US require students to complete some type of showcase portfolio, to our knowledge Western Washington University is unique in requiring a comprehensive standards-based capstone portfolio as a benchmark assessment for program completion.
  7. 7. HSP 304- Portfolio Learning <ul><li>Introduction to Portfolio Learning (HSP 304) is taken during the first quarter in the program and provides: </li></ul><ul><ul><ul><li>Overview of the HSP Curriculum. </li></ul></ul></ul><ul><ul><ul><li>Introduction to CSHSE National Standards and program specific outcomes (technology, information literacy, critical thinking, and writing proficiency. </li></ul></ul></ul><ul><ul><ul><li>Orientation to portfolio organization and design with all students receiving the Student Guide to Capstone Portfolio Process to support learning. </li></ul></ul></ul>
  8. 8. Portfolio Development Process <ul><ul><li>Instructors work with course plans that are aligned with CSHSE standards. Course plans typically identify at least one assignment per course appropriate for inclusion in the capstone portfolio. </li></ul></ul><ul><ul><li>Over a two-year course of study students are asked to update their portfolios in each class to reflect learning gained through coursework and field study. </li></ul></ul>
  9. 9. Capstone Portfolio (HSP 495) <ul><ul><li>Prior to graduation students finalize and present portfolios in a culminating course: HSP 495-Capstone Portfolio </li></ul></ul><ul><ul><li>Portfolios can be paper, web, or CD-Rom based (or any combination) </li></ul></ul><ul><ul><li>Students address all CSHSE and program standards through written essays and evidence documented from course work and field study (see Student Guide, Appendix F) </li></ul></ul>
  10. 10. Completion
  11. 11. Public Presentation <ul><ul><li>Six sections of the course were offered during Spring 2008 (113 students enrolled) </li></ul></ul><ul><ul><li>Four of the six sections hosted a public display of portfolios (Everett and Bellingham) </li></ul></ul>
  12. 12. Spring 2008, Bellingham
  13. 13. Assessment <ul><ul><li>The portfolio is a high stakes exit assessment, necessitating reliable scoring methods </li></ul></ul><ul><ul><li>A department goal is to make sure we have an assessment tool and philosophy that reflects our diverse student body and variations in instructor’s pedagogical viewpoints. </li></ul></ul>
  14. 14. The Rubric <ul><ul><li>Contains seven subscales assessing National Standards and Program Standards (critical thinking, information literacy, technology, writing proficiency), portfolio organization and demonstration of professional self </li></ul></ul><ul><ul><li>Used across program locations and sections </li></ul></ul><ul><ul><li>Total score serves as a benchmark exit assessment for completion of the major. (Current passing score is 70%) </li></ul></ul><ul><ul><li>Sub-scores are included as a performance assessment in the Woodring Information System (WIS) tracking system </li></ul></ul>Handout
  15. 15. CAP Work to Date: <ul><ul><li>Reviewed portfolio rater reliability literature </li></ul></ul><ul><ul><li>Examined rater reliability of rubric used by WWU Human Services faculty in Bellingham, Everett, and Cyber (electronic portfolios) </li></ul></ul><ul><ul><li>Assessed student perceptions of portfolio learning (through written response and exit survey data) </li></ul></ul><ul><ul><li>Determined faculty perceptions of portfolio learning/teaching </li></ul></ul>
  16. 16. Key Findings from Literature: Portfolio Rater Reliability Studies <ul><ul><ul><li>Inter-rater reliability for published studies of portfolio assessments typically fall in the .55-.80 range (average reliability is approximately .64, marginal) </li></ul></ul></ul><ul><ul><ul><li>Greatest predictor of inter-rater reliability in medical student portfolios is “quality of reflection” accounting for 66% of variance (Driessen, 2006) </li></ul></ul></ul><ul><ul><ul><li>Reliable assessment must involve detailed criteria for the learner and the evaluator presented at the beginning of the learning experience (NCLRC, 2008) </li></ul></ul></ul><ul><ul><ul><li>Reliability increases with triangulation (inclusion of 3 or more pieces of evidence in a portfolio section) </li></ul></ul></ul><ul><ul><ul><li>Mixed findings regarding reliability when assessing own students portfolios. </li></ul></ul></ul>
  17. 17. <ul><li>Data includes: </li></ul><ul><ul><li>Instructor’s total scores and subscale portfolio scores from 103 students enrolled in five sections of HSP 495 (Spring 2008). </li></ul></ul><ul><ul><li>20% of portfolios from Bellingham, Everett, and Cyber sites were randomly selected for reliability coding (by 2 other raters). </li></ul></ul>Examining Reliability of HS Capstone Rubric
  18. 18. Table 1: Mean (SD) Rubric Scores by Section 9.83 (.58) 8.79 (1.79) 9.0 (1.2) 9.91 (.31) 8.68 (1.06) Professional Self (10pt) 9.58 (1.44) 6.96 (2.29) 9.73 (.72) 9.64 (.67) 8.83 (.97) Technology (10pt) 10 (0) 6.92 (2.4) 9.73 (.70) 9.55 (.82) 8.40 (2.11) Information Literacy (10pt) 9.5 (1.73) 7.0 (2.45) 8.0 (2.83) 9.27 (1.0) 8.68 (1.17) Critical Thinking (10pt) 10 (0) 7.29 (2.14) 9.27 (1.32) 9.36 (1.57) 8.92 (1.09) Writing Proficiency (10pt) 39.79 (.72) 34.29 (5.65) 35.75 (3.8) 37.9 (4.41) 34.53 (3.93) National Standards (40pt) 10 (0) 8.21 (1.6) 9.50 (.96) 9.55 (.82) 9.10 (.85) Portfolio Organization (10pt) #5 98.71 #4 79.46 #3 90.98 #2 95.18 #1 87.13 Section Mean Total n= 103, Includes Bellingham, Everett, and Bremerton sections
  19. 19. Rubric Reliability: Analysis of Results <ul><ul><li>21 randomly selected portfolios from 5 sections (12 Bellingham, 6 Everett, 4 Cyber) </li></ul></ul><ul><ul><li>2 raters scored all 21 portfolios on all 7 dimensions </li></ul></ul>
  20. 20. Intraclass Correlations Coefficient NS .491 Total Score   .009 .712 7. Professional Self   NS .275 6. Technology [Tied to ISTE standards]   NS .153 5. Information literacy [Tied to ALA standards]   NS .463 4. Critical Thinking   NS .259 3. Writing proficiency   NS .524 2. CSHSE National Standards [for HS Education]   NS .399 1. Organization P-value Intraclass Correlation Coefficient Dimension
  21. 21. Primary Concerns <ul><li>Unacceptable estimates of reliability given that this is a high stakes assessment </li></ul><ul><li>Lack of consistency in overall scoring across sections of the course (variation in both means and standard deviations across sections) </li></ul>
  22. 22. POSSIBLE EXPLANATIONS <ul><ul><li>First cohorts of students to complete capstone portfolio. </li></ul></ul><ul><ul><li>Raters and instructors had not cross trained. </li></ul></ul><ul><ul><li>Restricted range of scores pointing to need to solidify categories rather than compare scores </li></ul></ul><ul><ul><li>Students had the opportunity to receive instructor and peer feedback and revise portfolios prior to grading and cross rating </li></ul></ul>
  23. 23. Student Perceptions of Portfolio Learning <ul><li>Data: </li></ul><ul><ul><li>Written response surveys from 93 students </li></ul></ul><ul><ul><ul><li>Bellingham=57 </li></ul></ul></ul><ul><ul><ul><li>Everett= 24 </li></ul></ul></ul><ul><ul><ul><li>Bremerton= 12 </li></ul></ul></ul><ul><ul><li>Senior exit survey data from 112 students </li></ul></ul>
  24. 24. Student’s Written Responses Regarding Portfolio Learning <ul><li>Summary of Strengths: </li></ul><ul><ul><li>Comprehensive review and synthesis of two years in the program </li></ul></ul><ul><ul><li>The peer review process was effective for portfolio development/completion </li></ul></ul><ul><ul><li>A culminating project to be proud of </li></ul></ul><ul><ul><li>Introductory class in the first quarter is critical for preparation </li></ul></ul>
  25. 25. Summary of Student Concerns <ul><ul><li>Some 304/495 instructors did not understand all standards. </li></ul></ul><ul><ul><li>More communication between faculty needed for scoring. </li></ul></ul><ul><ul><li>Promote digital portfolios (too much paper). </li></ul></ul><ul><ul><li>Portfolio checkpoints are needed throughout the program. </li></ul></ul><ul><ul><li>The Introductory course did not effectively prepare students (HSP 304). </li></ul></ul><ul><ul><li>Excessive workload in HSP 495 (class time not valuable). </li></ul></ul><ul><ul><li>The HS Program is too focused on standards. </li></ul></ul><ul><ul><li>The portfolio is not valuable for career related purposes. </li></ul></ul><ul><ul><li>Provide sample portfolios. </li></ul></ul>
  26. 26. Student Exit Survey Results * * Includes 112 exiting seniors, Spring 2008. Statement Agree Disagree My capstone portfolio demonstrates that I meet CSHSE Standards. 56% 45% Creating a Capstone Portfolio was a valuable part of my learning experience. 27% 45% Overall: Satisfaction with HS quality of course content. 91% 9%
  27. 27. Instructor Experience with Portfolio Learning and Teaching <ul><li>Includes: </li></ul><ul><ul><li>Written notes from group meeting with 4 of 6 instructors (HSP 495) </li></ul></ul>
  28. 28. Findings: Instructor’s Experiences with Portfolio Learning (4 of 6 faculty) <ul><li>Limitations </li></ul><ul><ul><li>Most students didn’t keep up with portfolios after their introductory course (no check points) </li></ul></ul><ul><ul><li>Extensive work for students and instructors </li></ul></ul><ul><ul><li>Class time is not valuable </li></ul></ul><ul><ul><li>Are students learning or just compiling? (Should we be charging tuition for this kind of course?) </li></ul></ul><ul><ul><li>Too much paper (make more use of CD-Roms) </li></ul></ul><ul><ul><li>Class doesn’t “feel good”. It seems arbitrary </li></ul></ul>
  29. 29. Findings : Instructor’s Experiences with Portfolio Learning <ul><li>Strengths </li></ul><ul><ul><li>Discussion and peer review was very helpful. </li></ul></ul><ul><ul><li>Final showcase event in Bellingham and Everett was a good culminating event. </li></ul></ul><ul><ul><li>Students initially complained about quantity of writing, but many thanked instructors afterwards. </li></ul></ul><ul><ul><li>Capstone learning has potential, but we are not “there” yet. </li></ul></ul>
  30. 30. Continuous Improvement… <ul><li>Summer/Fall 2008 </li></ul><ul><ul><ul><li>Revised course plan assignments and approach to teaching for the Intro. to Portfolio course, HSP 304. </li></ul></ul></ul><ul><ul><ul><li>Revised capstone portfolio grading rubric. </li></ul></ul></ul><ul><ul><ul><li>Changed some portfolio content from mandatory to optional. </li></ul></ul></ul><ul><ul><ul><li>Increased number of full time faculty teaching the Portfolio series. </li></ul></ul></ul><ul><ul><ul><li>Instructors shared examples of exemplary portfolios with students. </li></ul></ul></ul>
  31. 31. Continuous Improvement <ul><ul><li>Future Directions </li></ul></ul><ul><ul><li>Solidify categories and assign points to rubric. </li></ul></ul><ul><ul><li>Reanalyze reliability data to identify a Discrepancy Index. (Percentage of time raters score the same, are one category apart, etc…) </li></ul></ul><ul><ul><li>Film students discussing portfolio learning process and share with future groups. </li></ul></ul><ul><ul><li>Determine how to make most of class time (consider a small group tutorial model). </li></ul></ul><ul><ul><li>Continue to collect and analyze data from students (written responses and exit survey) during Spring, 2009 </li></ul></ul>
  32. 32. Future Directions - (Cont.) <ul><ul><li>Prior to Scoring : Systematize rater calibration process across all sites. </li></ul></ul><ul><ul><li>During Scoring : Engage in systematic dialogue among instructors while making scoring decisions. Ensure that irrelevant criteria is not being considered in the scores </li></ul></ul><ul><ul><li>Share and discuss findings of CAP with HSR department and other interested faculty (Winter, 2009) </li></ul></ul>
  33. 33. “ A portfolio tells a story. It is the story of knowing. Knowing about things... Knowing oneself... Knowing an audience... Students prove what they know with samples of their work.” (Paulson & Paulson, 1991, p.2)
  34. 34. Resources <ul><ul><ul><ul><li>Baume, D. & Yorke, M. (2002) The reliability of assessment by portfolio on a course to develop and accredit teachers in higher education, Studies in Higher Education, 27 , 7–25. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Driessen E, van Tartwijk J, van der Vleuten C, Wass V. (2007). Portfolios in medical education: why do they meet with mixed success? A systematic review. Medical Education, 41, 1224-1233. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Driessen, E. W., Overeem, K., Van Tartwijk, J., Van Der Vleuten, C., & Muijtjens, A. (2006). Validity of portfolio assessment: Which qualities determine ratings ? Medical Education , 40 (9), 862-866. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio: Principles for Practice Theory and Research . Cresskill, NJ: Hampton Press. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Johnston, B. (2004). Summative Assessment of Portfolios: An examination of different approaches to agreement over outcomes. Studies in Higher Education . 29 (3), 395-412. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Kimbell, R., & Stables, K. (2008). Researching design learning: Issues and findings from two decades of research and development . NY: Springer. </li></ul></ul></ul></ul>
  35. 35. Resources (Cont.) <ul><ul><ul><ul><li>McMullan M, Endacott R, Gray MA, Jasper M, Miller CML, Scholes J, Webb C. Portfolios and assessment of competence: a review of the literature. Journal of Advanced Nursing 2003; 41(3), 283-294. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>McNamara, T. L. & Bailey, R. (2006). Faculty/Staff perceptions of a standards-based exit portfolio system for gradate students. Innovative Higher Education, 3 (2), 129-141. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>National Capital Language Resource Center (2008). Portfolio Assessment in the Foreign Language Classroom. http:// </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment in an American college. Assessing writing, 10 , 61-73. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Pitts, J., Coles, C. & Thomas, P. (1999) Educational portfolios in the assessment of general practice trainers: Reliability of assessors, Medical Education, 33 , 515–520. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice , 10-24. </li></ul></ul></ul></ul>