UC Dual Pilot Study


Published on

Presentation for the Ohio Student Success Assessment Summit. Presentation done by Gigi Escoe.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Short identification of our study
  • Highlights of the methodology of our study
  • Note here the modifications that we’ve already made in the AAC&U metarubrics, trying to stay close to them but adapting them to this particular set of assignments
  • SurveyMonkey survey: 1. Please rate your overall impression of the validity of the Collegiate Learning Assessment(CLA)in providing feedback about students’ abilities in such areas as critical thinking, problem solving, and effective communication:
  • Review concerns that we’ve probably already mentioned; pull together main remaining concerns
  • UC Dual Pilot Study

    1. 1. “ Dual Pilot” study at University of Cincinnati: <ul><li>To consider the applicability of learning portfolios (and especially e-portfolios) alongside the CLA among a group of first-year Honors students </li></ul><ul><li>http://tinyurl.com/uceportfolioproject </li></ul>
    2. 2. Methodology: <ul><li>Study cohort of 120 first year Honors students </li></ul><ul><ul><li>Average ACT is 31 </li></ul></ul><ul><ul><li>Representing a range of UC colleges and programs </li></ul></ul><ul><ul><li>Briefed on experiment </li></ul></ul><ul><ul><li>Follow cohort over four-year time frame </li></ul></ul><ul><li>Administer the CLA as a course requirement </li></ul><ul><ul><li>Half complete “make an argument” and “critique an argument,” with the other half completing the problem-solving “performance task” </li></ul></ul><ul><ul><li>Student post-test survey also administered at testing site </li></ul></ul><ul><li>During the same month, students complete assignments designed to measure critical thinking and written communication </li></ul><ul><ul><li>Graded by faculty using a variation of AAC&U meta-rubrics </li></ul></ul><ul><li>, </li></ul>
    3. 3. Assessment of essays in the e-portfolios <ul><li>Variation on AAC&U (VALUE) meta-rubrics </li></ul><ul><ul><li>Critical Thinking </li></ul></ul><ul><ul><li>Written Communication </li></ul></ul>
    4. 4. Preliminary results – CLA/VALUE score correlations <ul><li>CLA report delayed 8 months, returning data to students problematic </li></ul><ul><li>R-square between the Performance Task Score and Scaled SAT/ACT score:  0.053 (correlation= 0.23) </li></ul><ul><li>R-square between the Analytical Writing Score and Scaled SAT/ACT score: 0.059 (correlation = 0.24) </li></ul><ul><li>Ceiling effects are a concern: honors student unadjusted percentile score was 98 th percentile, adjusted for SAT 95 th; VALUE rubrics may be calibrated to avoid ceiling effects. </li></ul>
    5. 5. Some Findings from the Student Survey <ul><ul><li>Usefulness of the test (waste of time or only mildly useful=50%,useful= 40%, very useful=10%) </li></ul></ul><ul><ul><li>Describe the test experience (unpleasant=37%, indifferent 48%, pleasant= 15%) </li></ul></ul><ul><ul><li>Effort exerted (little or none 8%, some 40%, a great deal 52%) </li></ul></ul><ul><ul><li>Incentives to take the test seriously (would do it for free 8%, self-improvement 39%, financial incentive required- 43%) </li></ul></ul><ul><ul><li>Experiences that prepared students to do well on the test: prep courses for ACT and SAT </li></ul></ul>
    6. 6. UC’s concerns about the CLA: <ul><li>Questionable reliability and validity of results </li></ul><ul><li>Lack of integration into the curriculum </li></ul><ul><li>Disconnect between the CLA and UC’s mission and students </li></ul><ul><li>Absence of ownership by the faculty </li></ul><ul><li>Question motivation of students taking the test if it is not embedded in a course; embedding is difficult as CLA is designed with the institution, not the individual, as the “unit” of analysis </li></ul>
    7. 7. Future Plans <ul><li>Continue to pilot embedding CLA in FYE courses for freshmen; capstone courses for seniors </li></ul><ul><li>Working with IR to create reasonably representative samples </li></ul><ul><li>Include faculty in conversation and find ways to use the data to provide feedback to students and programs </li></ul><ul><li>Explore other measures of general education outcomes: </li></ul><ul><ul><li>Capstone rubric assessment </li></ul></ul><ul><ul><li>Expanded use of e-portfolios </li></ul></ul><ul><ul><li>Data from experiential learning supervisors through UC’s </li></ul></ul><ul><ul><li>I-LEAP project </li></ul></ul>