“ Dual Pilot” study at University of Cincinnati: To consider the applicability of learning portfolios (and especially e-portfolios) alongside the CLA among a group of first-year Honors students http://tinyurl.com/uceportfolioproject

To consider the applicability of learning portfolios (and especially e-portfolios) alongside the CLA among a group of first-year Honors students

http://tinyurl.com/uceportfolioproject

Methodology: Study cohort of 120 first year Honors students Average ACT is 31 Representing a range of UC colleges and programs Briefed on experiment Follow cohort over four-year time frame Administer the CLA as a course requirement Half complete “make an argument” and “critique an argument,” with the other half completing the problem-solving “performance task” Student post-test survey also administered at testing site During the same month, students complete assignments designed to measure critical thinking and written communication Graded by faculty using a variation of AAC&U meta-rubrics ,

Study cohort of 120 first year Honors students

Average ACT is 31

Representing a range of UC colleges and programs

Briefed on experiment

Follow cohort over four-year time frame

Administer the CLA as a course requirement

Half complete “make an argument” and “critique an argument,” with the other half completing the problem-solving “performance task”

Student post-test survey also administered at testing site

During the same month, students complete assignments designed to measure critical thinking and written communication

Graded by faculty using a variation of AAC&U meta-rubrics

,

Assessment of essays in the e-portfolios Variation on AAC&U (VALUE) meta-rubrics Critical Thinking Written Communication

Variation on AAC&U (VALUE) meta-rubrics

Critical Thinking

Written Communication

Preliminary results – CLA/VALUE score correlations CLA report delayed 8 months, returning data to students problematic R-square between the Performance Task Score and Scaled SAT/ACT score: 0.053 (correlation= 0.23) R-square between the Analytical Writing Score and Scaled SAT/ACT score: 0.059 (correlation = 0.24) Ceiling effects are a concern: honors student unadjusted percentile score was 98 th percentile, adjusted for SAT 95 th; VALUE rubrics may be calibrated to avoid ceiling effects.

CLA report delayed 8 months, returning data to students problematic

R-square between the Performance Task Score and Scaled SAT/ACT score: 0.053 (correlation= 0.23)

R-square between the Analytical Writing Score and Scaled SAT/ACT score: 0.059 (correlation = 0.24)

Ceiling effects are a concern: honors student unadjusted percentile score was 98 th percentile, adjusted for SAT 95 th; VALUE rubrics may be calibrated to avoid ceiling effects.

Some Findings from the Student Survey Usefulness of the test (waste of time or only mildly useful=50%,useful= 40%, very useful=10%) Describe the test experience (unpleasant=37%, indifferent 48%, pleasant= 15%) Effort exerted (little or none 8%, some 40%, a great deal 52%) Incentives to take the test seriously (would do it for free 8%, self-improvement 39%, financial incentive required- 43%) Experiences that prepared students to do well on the test: prep courses for ACT and SAT

Usefulness of the test (waste of time or only mildly useful=50%,useful= 40%, very useful=10%)

Describe the test experience (unpleasant=37%, indifferent 48%, pleasant= 15%)

Effort exerted (little or none 8%, some 40%, a great deal 52%)

Incentives to take the test seriously (would do it for free 8%, self-improvement 39%, financial incentive required- 43%)

Experiences that prepared students to do well on the test: prep courses for ACT and SAT

UC’s concerns about the CLA: Questionable reliability and validity of results Lack of integration into the curriculum Disconnect between the CLA and UC’s mission and students Absence of ownership by the faculty Question motivation of students taking the test if it is not embedded in a course; embedding is difficult as CLA is designed with the institution, not the individual, as the “unit” of analysis

Questionable reliability and validity of results

Lack of integration into the curriculum

Disconnect between the CLA and UC’s mission and students

Absence of ownership by the faculty

Question motivation of students taking the test if it is not embedded in a course; embedding is difficult as CLA is designed with the institution, not the individual, as the “unit” of analysis

Future Plans Continue to pilot embedding CLA in FYE courses for freshmen; capstone courses for seniors Working with IR to create reasonably representative samples Include faculty in conversation and find ways to use the data to provide feedback to students and programs Explore other measures of general education outcomes: Capstone rubric assessment Expanded use of e-portfolios Data from experiential learning supervisors through UC’s I-LEAP project

Continue to pilot embedding CLA in FYE courses for freshmen; capstone courses for seniors

Working with IR to create reasonably representative samples

Include faculty in conversation and find ways to use the data to provide feedback to students and programs

Explore other measures of general education outcomes:

Capstone rubric assessment

Expanded use of e-portfolios

Data from experiential learning supervisors through UC’s

I-LEAP project