The University of Cincinnati conducted a "Dual Pilot" study to evaluate the use of learning portfolios and the CLA among first-year Honors students. 120 students took the CLA and completed writing assignments assessed with AAC&U rubrics. Preliminary results showed low correlations between CLA scores and SAT/ACT scores, and ceiling effects were a concern with honors students. A student survey found that only 10% found the CLA very useful while 37% found the testing experience unpleasant. The university had concerns about the reliability and validity of CLA results as well as its disconnect from the curriculum and mission. Future plans include embedding the CLA in more courses and exploring other assessment measures.

1. “ Dual Pilot” study at University of Cincinnati: To consider the applicability of learning portfolios (and especially e-portfolios) alongside the CLA among a group of first-year Honors students http://tinyurl.com/uceportfolioproject

2. Methodology: Study cohort of 120 first year Honors students Average ACT is 31 Representing a range of UC colleges and programs Briefed on experiment Follow cohort over four-year time frame Administer the CLA as a course requirement Half complete “make an argument” and “critique an argument,” with the other half completing the problem-solving “performance task” Student post-test survey also administered at testing site During the same month, students complete assignments designed to measure critical thinking and written communication Graded by faculty using a variation of AAC&U meta-rubrics ,

3. Assessment of essays in the e-portfolios Variation on AAC&U (VALUE) meta-rubrics Critical Thinking Written Communication

4. Preliminary results – CLA/VALUE score correlations CLA report delayed 8 months, returning data to students problematic R-square between the Performance Task Score and Scaled SAT/ACT score: 0.053 (correlation= 0.23) R-square between the Analytical Writing Score and Scaled SAT/ACT score: 0.059 (correlation = 0.24) Ceiling effects are a concern: honors student unadjusted percentile score was 98 th percentile, adjusted for SAT 95 th; VALUE rubrics may be calibrated to avoid ceiling effects.

5. Some Findings from the Student Survey Usefulness of the test (waste of time or only mildly useful=50%,useful= 40%, very useful=10%) Describe the test experience (unpleasant=37%, indifferent 48%, pleasant= 15%) Effort exerted (little or none 8%, some 40%, a great deal 52%) Incentives to take the test seriously (would do it for free 8%, self-improvement 39%, financial incentive required- 43%) Experiences that prepared students to do well on the test: prep courses for ACT and SAT

6. UC’s concerns about the CLA: Questionable reliability and validity of results Lack of integration into the curriculum Disconnect between the CLA and UC’s mission and students Absence of ownership by the faculty Question motivation of students taking the test if it is not embedded in a course; embedding is difficult as CLA is designed with the institution, not the individual, as the “unit” of analysis

7. Future Plans Continue to pilot embedding CLA in FYE courses for freshmen; capstone courses for seniors Working with IR to create reasonably representative samples Include faculty in conversation and find ways to use the data to provide feedback to students and programs Explore other measures of general education outcomes: Capstone rubric assessment Expanded use of e-portfolios Data from experiential learning supervisors through UC’s I-LEAP project