Rubrics and Metarubrics from the AAC&U VALUE Project Mark Nicholas & Rich Robles University of Cincinnati
“ Dual Pilot” study at  University of Cincinnati <ul><li>To consider the applicability of learning portfolios (and especia...
Written Communication Rubric Changes <ul><li>Expanded a four scale to five (to provide a mid point) </li></ul><ul><li>Simp...
Written Communication Rubric Challenges <ul><li>Inter-rater reliability (r = 0.6964) </li></ul><ul><ul><li>Easier to asses...
Assessing Critical Thinking (CT) <ul><li>Nature of Artifact </li></ul><ul><ul><li>Proposal for honors experience </li></ul...
CT Rubric Changes <ul><li>Changed the scale from six to five (to provide a mid point) </li></ul><ul><li>Included the dispo...
CT as a construct <ul><li>An esoteric concept </li></ul><ul><ul><li>Skills and dispositions </li></ul></ul><ul><li>Epistem...
CT Rubric Challenges <ul><ul><li>Articulation of the definition that underlies the rubric </li></ul></ul><ul><ul><li>Diffe...
CT Rubric Challenges <ul><li>Need to develop guidelines on the nature of artifacts </li></ul><ul><ul><li>Does the artifact...
Metarubric vs. CLA <ul><li>Inter-rater reliability </li></ul><ul><ul><li>Inconsistency in the evaluators’ judgments </li><...
UC Dual Pilot Study http://tinyurl.com/uceportfolioproject
AAC&U VALUE Metarubrics at UC
Upcoming SlideShare
Loading in...5
×

AAC&U VALUE Metarubrics at UC

610

Published on

Presentation for the Ohio Student Success Assessment Summit. Presentation done by Rich Robles and Mark Nicholas

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
610
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "AAC&U VALUE Metarubrics at UC"

  1. 1. Rubrics and Metarubrics from the AAC&U VALUE Project Mark Nicholas & Rich Robles University of Cincinnati
  2. 2. “ Dual Pilot” study at University of Cincinnati <ul><li>To consider the applicability of learning portfolios (and especially e-portfolios) alongside the CLA among a group of first-year Honors students </li></ul><ul><li>Applied the written communication and critical thinking metarubrics </li></ul>
  3. 3. Written Communication Rubric Changes <ul><li>Expanded a four scale to five (to provide a mid point) </li></ul><ul><li>Simplified terminology </li></ul><ul><li>Changed text in each criteria to make each element distinct as we felt an overlap </li></ul>
  4. 4. Written Communication Rubric Challenges <ul><li>Inter-rater reliability (r = 0.6964) </li></ul><ul><ul><li>Easier to assess as most people know or is aware of what is good writing </li></ul></ul><ul><ul><li>Can be assessed outside the context of the nature of a specific artifact or discipline in a limited sense </li></ul></ul><ul><li>Had an expert on the team that guided the evaluation process </li></ul>
  5. 5. Assessing Critical Thinking (CT) <ul><li>Nature of Artifact </li></ul><ul><ul><li>Proposal for honors experience </li></ul></ul><ul><ul><li>Equivalent to a proposal for an experiment </li></ul></ul><ul><ul><li>Part of assigned homework </li></ul></ul>
  6. 6. CT Rubric Changes <ul><li>Changed the scale from six to five (to provide a mid point) </li></ul><ul><li>Included the dispositional element of open-mindedness </li></ul><ul><li>Changed text in criteria and scale to make each element distinct as we felt an overlap </li></ul><ul><li>Tweaked the rubric to the nature of the artifact to which it was applied </li></ul>
  7. 7. CT as a construct <ul><li>An esoteric concept </li></ul><ul><ul><li>Skills and dispositions </li></ul></ul><ul><li>Epistemic skilled based Vs critical philosophical approach </li></ul><ul><ul><li>Impacts conceptualization </li></ul></ul><ul><ul><li>Impacts assessment </li></ul></ul><ul><ul><li>Interpretation of assessment </li></ul></ul><ul><li>Discipline general and discipline specific CT </li></ul>
  8. 8. CT Rubric Challenges <ul><ul><li>Articulation of the definition that underlies the rubric </li></ul></ul><ul><ul><li>Differing definitions and a lack of definition among faculty (Paul, Elder & Bartell, 1997*) </li></ul></ul><ul><ul><ul><li>Disciplinary interpretations (example) </li></ul></ul></ul><ul><ul><li>The need to </li></ul></ul><ul><ul><ul><li>Orient all evaluators on a conceptualization </li></ul></ul></ul><ul><ul><ul><li>Interpret results within the context of the definition and its limitations </li></ul></ul></ul>* Paul, R., Elder, L., & Bartell, T. (1997). California teacher preparation for instruction in critical thinking: Research findings and policy recommendations. California: Commission on Teacher Credentialing.
  9. 9. CT Rubric Challenges <ul><li>Need to develop guidelines on the nature of artifacts </li></ul><ul><ul><li>Does the artifact and its content lend itself to the assessment of CT? (example) </li></ul></ul><ul><li>Inter-rater reliability (still in progress) </li></ul><ul><li>Apply the rubric twice </li></ul><ul><ul><li>on the proposal for the honors experience and </li></ul></ul><ul><ul><li>after the experience </li></ul></ul>
  10. 10. Metarubric vs. CLA <ul><li>Inter-rater reliability </li></ul><ul><ul><li>Inconsistency in the evaluators’ judgments </li></ul></ul><ul><ul><li>Norming sessions </li></ul></ul><ul><ul><li>Differing perspectives </li></ul></ul><ul><li>Test value </li></ul><ul><ul><li>CLA (voluntary) vs Metarubric (homework) </li></ul></ul><ul><li>Contextual </li></ul><ul><ul><li>Both CLA and metarubric provides scope for context which is important for assessing writing and CT </li></ul></ul><ul><li>Time scales </li></ul><ul><ul><li>Grading using rubrics is time consuming </li></ul></ul><ul><ul><li>CLA open ended (eight month turnaround) </li></ul></ul>
  11. 11. UC Dual Pilot Study http://tinyurl.com/uceportfolioproject
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×