RUBRIC DEVELOPMENT
Why use
rubrics?
Characteristics
of good rubrics
Definition &
types of rubric
Strategies for
rubric
development
SOLO taxonomy
Phenomeno-
graphic sorting
Using rubrics
with students
 Definition
 A scoring tool that lays out the specific expectations for an
assessment task (Stevens & Levi, 2005)
 A set of clear explanations or criteria used to help teachers and
students focus on what is valued in a subject, topic, or activity
(Russell, & Airasian, 2012).
 Components of a rubric:
 Criteria/Indicator
 aspects of an assessment task which the assessor takes into account
when making their judgment
 May use different weightings for different criteria
 Level of Attainment
 often use grade level descriptors
 Types of rubrics:
 Descriptive (Analytic), Holistic
 Why and when we use particular types of rubrics
SCORING RUBRIC
WHY USE RUBRICS?
For teachers:
• Prompt a criterion-referenced
assessment
• Provide students with detailed
and timely feedback
• Encourage critical thinking
• Facilitate communication with
others involved in scoring
• Help to refine teaching
skills/learning activities
For students:
• Clarify the teacher’s
expectations of student
performance
• Provide informative
descriptions of expected
performance
• Help to monitor and critique
own work
 Well defined
 Clearly describe the expected level of student performance for
each criterion in a rubric
 Avoid general evaluative words (poor, excellent, etc.)
 Use specific objective terms (correctly identifies, uses only
basic vocabulary, chooses incorrect formula… )
 Context specific
 Describe what teachers expect from student for a given
performance or work product on a particular subject domain
 Viable for instruction
CHARACTERISTICS OF GOOD RUBRICS (1)
 Finite and exhaustive
 Every response must be scorable
 Too many score levels is confusing for students and causes
disagreement among teacher scores
 Ordered
 Represent the different levels of learning targets as defined by
LP
 Related to Common Core theme/strand
 Coherent with the cognitive complexity of the standards
CHARACTERISTICS OF GOOD RUBRICS (2)
Components:
(1) Task
description
(2) Assessment
criteria
(3) Performance
levels
Task Description:
Criteria
1
Criteria
2
Criteria
3
Criteria
4
Total
Level 5
Level 4
Level 3
Level 2
Level 1
ANALYTIC (DESCRIPTIVE) RUBRICS
Advantage:
 Provides
judgment on
each criterion
Disadvantage:
 Time consuming
to make
Task Description:
Criteria
Level 5 Overall description of Level 5
Level 4 Overall description of Level 4
Level 3 Overall description of Level 3
Level 2 Overall description of Level 2
Level 1 Overall description of Level 1
HOLISTIC RUBRICS
 A single scale with all criteria to be included in the
evaluation being considered together
 Based on an overall judgment of student work
Advantage:
 Saves time in
developing and
scoring
Disadvantage:
 Does not provide
specific feedback
for improvement
 On performance-based tasks:
 extended response items
 projects
 presentations
 portfolios
WHEN TO USE RUBRICS
 Reflecting on the task & content
 Learning outcomes of the unit and the particular assessment
 What we want from the students, why we created this
assessment, what our expectations are
 Listing the learning outcomes & expectations
 Focus on the particular details of the task and what specific
learning objectives we expect to see in the completed task
 Grouping & labeling the outcomes & criteria
 Organize the results of reflections, group similar expectations
together to become the rubric Indicators
 Application of a rubric format
 Apply the templates & descriptions to the final form of the
rubrics
4 KEY STAGES IN CREATING RUBRICS
 Relationship between the intended Learning
Progression (LP) and rubrics
 Direct use of the LP as standard reference
 Use of a general strategy (i.e. modified Bloom’s
taxonomy or SOLO taxonomy)
 Assign the target outcome as mapped onto the Cognitive Rigor
Matrix as the maximum level of the rubric
 Use the SOLO taxonomy as scoring
 Phenomenographic sorting
STRATEGIES FOR RUBRIC DEVELOPMENT
SOLO TAXONOMY
Possible
Score
Level Responses that …
4 Extended not only include all relevant pieces of information, but also
extend the response to integrate relevant pieces of
information not in the stimulus.
3 Relational integrate all relevant pieces of information from the stimulus.
2 Multistructural respond to several relevant pieces of information from the
stimulus.
1 Unistructural respond to only one relevant piece of information from the
stimulus.
0 Pre-structural consist only of irrelevant information.
* Modified from Wilson (2005, p. 75)
 A possible value of 0 – 4 can be used to score each
question
1. Sort student responses into performance level piles
(excellent, good, …, and poor), or in terms of the
levels of understanding of the responses
2. Describe similarities within a pile and differences
between
 These similarities and differences can inform the different
level of rubrics
3. Do sorting in pairs
 To reconfirm matches & mismatches
 Allow dialogue to maximize the effectiveness of rubric
development
PHENOMENOGRAPHIC SORTING
 Explain what the test will emphasize
 Emphasis and expectations will be delineated in the
assessment criteria in the rubrics
 Inform students how the assessment will be scored
 Explain what each of the assessment criteria defined in the
rubrics means
 Explain how the results will be used
 Explain the importance of the test scores in the student’s
learning progression
USING RUBRICS WITH STUDENTS
 Make sure that the wording of the rubrics is
understandable to students (simplify wording for
lower grades).
 Works best with holistic rubrics (or with a combined
version of analytic descriptions).
 Provide rubrics to students in advance of the
administration of the assessment.
 Alternatively, students can be provided with a
general rubric and a small number of papers (names
removed). Students can score the papers in small
groups; groups are required to come to consensus on
the grade to be assigned.
HOW TO USE RUBRICS WITH STUDENTS
 Nitko, A. J., & Brookhart, S. (2007). Educational assessment of students. Upper
Saddle River, NJ: Pearson Education, Inc.
 McMillan, J. H. (2007). Classroom assessment. Principles and practice for
effective standard-based instruction (4th ed.). Boston: Pearson - Allyn & Bacon.
 Oregon Department of Education. (2014, June). Assessment guidance.
 Popham, W. J. (2014). Criterion-referenced measurement: A half-century wasted?
Paper presented at the Annual Meeting of National Council on Measurement in
Education, Philadephia, PA .
 Popham, W. J. (2014). Classroom assessment: What teachers needs to know. San
Francisco, CA: Pearson
 Russell, M. K., & Airasian, P. W. (2012). Classroom assessment: Concepts and
applications. New York, NY: McGraw -Hill.
 Stevens, D. & Levi, A. (2005). Introduction to rubrics. As assessment tool to save
grading time, convey effective feedback, and promote student learning . Sterling:
Stylus Publishing, LLC
 Wihardini, D. (2010). Assessment development II. Unpublished manuscript.
Research and Development Department, Binus Business School, Jakarta,
Indonesia.
 Wilson, M. (2005). Constructing measures: An item response modeling approach.
Mahwah, NJ: Lawrence Erlbaum Associates.
BIBLIOGRAPHY
Rubrics PPT by the Oregon Department of Education and Berkeley Evaluation and
Assessment Research Center is licensed under a CC BY 4.0.
Y ou a re f ree t o:
 S hare — copy and redistribute the material in any medium or format
 Adapt — remix, transform, and build upon the material
U nder t he fo llowing t erms:
 Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes
were made. You may do so in any reasonable manner, but not in any way that suggests the licensor
endorses you or your use.
 N onCommercial — You may not use the material for commercial purposes .
 S hareAlike — If you remix, transform, or build upon the material, you must distribute your
contributions under the same license as the original.
Oregon Department of Education welcomes editing of these resources and would
greatly appreciate being able to learn from the changes made. To share an edited
version of this resource, please contact Cristen McLean, cristen.mclean@state.or.us.
CREATIVE COMMONS LICENSE

rubric_development.pptx

  • 1.
    RUBRIC DEVELOPMENT Why use rubrics? Characteristics ofgood rubrics Definition & types of rubric Strategies for rubric development SOLO taxonomy Phenomeno- graphic sorting Using rubrics with students
  • 2.
     Definition  Ascoring tool that lays out the specific expectations for an assessment task (Stevens & Levi, 2005)  A set of clear explanations or criteria used to help teachers and students focus on what is valued in a subject, topic, or activity (Russell, & Airasian, 2012).  Components of a rubric:  Criteria/Indicator  aspects of an assessment task which the assessor takes into account when making their judgment  May use different weightings for different criteria  Level of Attainment  often use grade level descriptors  Types of rubrics:  Descriptive (Analytic), Holistic  Why and when we use particular types of rubrics SCORING RUBRIC
  • 3.
    WHY USE RUBRICS? Forteachers: • Prompt a criterion-referenced assessment • Provide students with detailed and timely feedback • Encourage critical thinking • Facilitate communication with others involved in scoring • Help to refine teaching skills/learning activities For students: • Clarify the teacher’s expectations of student performance • Provide informative descriptions of expected performance • Help to monitor and critique own work
  • 4.
     Well defined Clearly describe the expected level of student performance for each criterion in a rubric  Avoid general evaluative words (poor, excellent, etc.)  Use specific objective terms (correctly identifies, uses only basic vocabulary, chooses incorrect formula… )  Context specific  Describe what teachers expect from student for a given performance or work product on a particular subject domain  Viable for instruction CHARACTERISTICS OF GOOD RUBRICS (1)
  • 5.
     Finite andexhaustive  Every response must be scorable  Too many score levels is confusing for students and causes disagreement among teacher scores  Ordered  Represent the different levels of learning targets as defined by LP  Related to Common Core theme/strand  Coherent with the cognitive complexity of the standards CHARACTERISTICS OF GOOD RUBRICS (2)
  • 6.
    Components: (1) Task description (2) Assessment criteria (3)Performance levels Task Description: Criteria 1 Criteria 2 Criteria 3 Criteria 4 Total Level 5 Level 4 Level 3 Level 2 Level 1 ANALYTIC (DESCRIPTIVE) RUBRICS Advantage:  Provides judgment on each criterion Disadvantage:  Time consuming to make
  • 7.
    Task Description: Criteria Level 5Overall description of Level 5 Level 4 Overall description of Level 4 Level 3 Overall description of Level 3 Level 2 Overall description of Level 2 Level 1 Overall description of Level 1 HOLISTIC RUBRICS  A single scale with all criteria to be included in the evaluation being considered together  Based on an overall judgment of student work Advantage:  Saves time in developing and scoring Disadvantage:  Does not provide specific feedback for improvement
  • 8.
     On performance-basedtasks:  extended response items  projects  presentations  portfolios WHEN TO USE RUBRICS
  • 9.
     Reflecting onthe task & content  Learning outcomes of the unit and the particular assessment  What we want from the students, why we created this assessment, what our expectations are  Listing the learning outcomes & expectations  Focus on the particular details of the task and what specific learning objectives we expect to see in the completed task  Grouping & labeling the outcomes & criteria  Organize the results of reflections, group similar expectations together to become the rubric Indicators  Application of a rubric format  Apply the templates & descriptions to the final form of the rubrics 4 KEY STAGES IN CREATING RUBRICS
  • 10.
     Relationship betweenthe intended Learning Progression (LP) and rubrics  Direct use of the LP as standard reference  Use of a general strategy (i.e. modified Bloom’s taxonomy or SOLO taxonomy)  Assign the target outcome as mapped onto the Cognitive Rigor Matrix as the maximum level of the rubric  Use the SOLO taxonomy as scoring  Phenomenographic sorting STRATEGIES FOR RUBRIC DEVELOPMENT
  • 11.
    SOLO TAXONOMY Possible Score Level Responsesthat … 4 Extended not only include all relevant pieces of information, but also extend the response to integrate relevant pieces of information not in the stimulus. 3 Relational integrate all relevant pieces of information from the stimulus. 2 Multistructural respond to several relevant pieces of information from the stimulus. 1 Unistructural respond to only one relevant piece of information from the stimulus. 0 Pre-structural consist only of irrelevant information. * Modified from Wilson (2005, p. 75)  A possible value of 0 – 4 can be used to score each question
  • 12.
    1. Sort studentresponses into performance level piles (excellent, good, …, and poor), or in terms of the levels of understanding of the responses 2. Describe similarities within a pile and differences between  These similarities and differences can inform the different level of rubrics 3. Do sorting in pairs  To reconfirm matches & mismatches  Allow dialogue to maximize the effectiveness of rubric development PHENOMENOGRAPHIC SORTING
  • 13.
     Explain whatthe test will emphasize  Emphasis and expectations will be delineated in the assessment criteria in the rubrics  Inform students how the assessment will be scored  Explain what each of the assessment criteria defined in the rubrics means  Explain how the results will be used  Explain the importance of the test scores in the student’s learning progression USING RUBRICS WITH STUDENTS
  • 14.
     Make surethat the wording of the rubrics is understandable to students (simplify wording for lower grades).  Works best with holistic rubrics (or with a combined version of analytic descriptions).  Provide rubrics to students in advance of the administration of the assessment.  Alternatively, students can be provided with a general rubric and a small number of papers (names removed). Students can score the papers in small groups; groups are required to come to consensus on the grade to be assigned. HOW TO USE RUBRICS WITH STUDENTS
  • 15.
     Nitko, A.J., & Brookhart, S. (2007). Educational assessment of students. Upper Saddle River, NJ: Pearson Education, Inc.  McMillan, J. H. (2007). Classroom assessment. Principles and practice for effective standard-based instruction (4th ed.). Boston: Pearson - Allyn & Bacon.  Oregon Department of Education. (2014, June). Assessment guidance.  Popham, W. J. (2014). Criterion-referenced measurement: A half-century wasted? Paper presented at the Annual Meeting of National Council on Measurement in Education, Philadephia, PA .  Popham, W. J. (2014). Classroom assessment: What teachers needs to know. San Francisco, CA: Pearson  Russell, M. K., & Airasian, P. W. (2012). Classroom assessment: Concepts and applications. New York, NY: McGraw -Hill.  Stevens, D. & Levi, A. (2005). Introduction to rubrics. As assessment tool to save grading time, convey effective feedback, and promote student learning . Sterling: Stylus Publishing, LLC  Wihardini, D. (2010). Assessment development II. Unpublished manuscript. Research and Development Department, Binus Business School, Jakarta, Indonesia.  Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates. BIBLIOGRAPHY
  • 16.
    Rubrics PPT bythe Oregon Department of Education and Berkeley Evaluation and Assessment Research Center is licensed under a CC BY 4.0. Y ou a re f ree t o:  S hare — copy and redistribute the material in any medium or format  Adapt — remix, transform, and build upon the material U nder t he fo llowing t erms:  Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.  N onCommercial — You may not use the material for commercial purposes .  S hareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. Oregon Department of Education welcomes editing of these resources and would greatly appreciate being able to learn from the changes made. To share an edited version of this resource, please contact Cristen McLean, cristen.mclean@state.or.us. CREATIVE COMMONS LICENSE

Editor's Notes

  • #2 This chapter will involve the characteristics of good rubrics, and some general strategies that can be used to produce them.
  • #3 Scoring rubrics involve The specific expectations of an assessment task A set of clear expectations or criteria, that allows both teachers and students to focus on what is valued in a subject, topic, or activity The components of a rubric may include Aspects of a task which may be taken into account when making a judgment about performance ont hat task Different weightings for different criteria Levels of attainment (which may include grade level descriptors) We’ll be talking in the coming pages about two types of rubrics: Descriptive or analytic Holistic As well as when and why we use particular types of rubrics
  • #4  Rubrics have a number of advantages for both teachers and students. For teachers They provide the criteria against which an assessment should be evaluated They assist in providing a student with detailed and timely feedback about performance They encourage the teacher’s critical thinking They facilitate communication with colleagues using the same rubric for scoring An, they provide feedback that can be used to refine teaching and learning activities For students They clarify the teacher’s expectations around assessment They provide information about the expected performance And, they can help students to monitor, and to critique, their own work based on the rubric levels
  • #5 Good rubrics have several characteristics. These include They are well defined They clearly define the expected student performance for each level They use specific objective terms involving behavior, rather than general evaluative terms (which each user of the rubric may define differently) They are context specific They define teacher expectations for students for given performances in particular subject domains This makes them useful for instruction as well as assessment purposes.
  • #6 Additional components include that They are finite No need to make a new scoring category for each new type of response – and every response must be scorable (even those that are off task, or flippant!) They are exhaustive There should be a place for every type of response – even irrelevant or flippant types of responses (I think of this as “a place for everything and everything in its place”) The number of levels is important; we want to differentiate between important level differences, but too many score levels can confuse both students and teachers They are (at least partially) ordered The order should represent the different levels of learning targets as represented by the LP They should be related to the common core theme or strand, and coherent with the cognitive complexity of the standards.
  • #7 The first of the two types of rubrics we’ll discuss is an analytic (or descriptive) type. Analytic rubrics allow scoring of a task on several different aspects (or criteria) of the task (for example, grammar and creativity in ELA, or graphical skills and algorithmic correctness in math) Components include the general task description, descriptions of the criteria, and specific performance level descriptions for each of the criteria. Such rubrics provide separate judgments for each criteria Good for diagnostic purposes Provides more information from larger tasks However, such rubrics can be time consuming to make, and to use for scoring purposes.
  • #8 The second type of rubric is a holistic rubric The holistic type provides a single scale in which all criteria for evaluation are considered together, providing an overall judgment of student work as a whole Such a rubric is quicker to make and use However, it does not provide diagnostic feedback for the student as to which components need improvement. It can also be tricky to use if a student is much better at one subskill than another
  • #9 Rubrics are most often used on performance-based tasks, such as extended response items, projects, presentations, or portfolios. They are of course not needed for forced-choice items (like multiple choice or true-false), and are rarely needed for short-answer items.
  • #10 There are several key steps to designing a rubric for a particular assessment task First, we need to think about the content, the task, and the students: What did we intend to teach? What is the purpose of the assessment? What we want from our students? Having done this, we need to list the particular details of the learning outcomes and objectives that we would like to see in an ideal response to a completed task. We then organize the results, grouping similar expectations together if we are designing an analytic rubric, and determining which expectations are easier and more difficult Finally, we use a rubric design template to produce the final form of the rubric
  • #11 We’ll now be discussing several strategies that can be used to design rubrics for a particular task. The first of these includes using the description of the Learning Progression levels (either within or between grades) as general descriptors to design the specific levels of the rubric. The second is the use of a general rubric strategy such as Bloom’s taxonomy or a modification of it, or the Structure of the Learning Outcome taxonomy (which we’ll show in the next slide), to design the specific levels of the rubric The third is a technique called phenomenographic sorting, which we’ll explore in more detail in a few minutes. This technique basically involves sorting a set of student outcomes according to quality, and using the result to describe the specific rubric levels.
  • #12 The SOLO (short for Structure of the Learning Outcome) taxonomy has five levels, which we’ve labeled 0 through 4. The lowest level, prestructural consists of only information that is irrelevant to the question The next highest level , unistructural, consists of only one relevant piece if information Level 2, multistructural, consists of multiple pieces of information, often presented as a list Level 3. Relational, consists of multiple pieces of information, integrated into a coherent response The highest level, Extended, extends the response to information not included or required by the stimulus
  • #13 Phenomenographic sorting involves sorting student responses to a particular item into piles of increasingly good performance (performances within a pile should be similar in level of quality, and piles should be ordered according to quality). Then, the differences between the piles should be described. These can be used to develop descriptions of different levels of the rubric It is often most effective to do the sorting in pairs, and discussing the similarities and differences with a partner
  • #14 Rubrics can also be used as a tool with students. This has a number of advantages They can be used to explain what the test will emphasize, and what the expectations will be They can be used to explain how the assessment will be scored, and what the meanings of the criteria are Finally, they can be used to explain how the results will be used
  • #15 How to go about using rubrics with students: First, make sure that the language of the rubric can be understood by students (you may have to simplify them for younger children). Holistic rubrics, or a compilation of the criteria for analytic rubrics, are easier for students to follow. They should provide characteristics of a good response, as opposed to information about correct answers It’s a good idea to provide the rubrics as handouts to students in advance of administering the assessment, as they can use them to focus their preparation. For take-home projects, rubrics can be included in the project packet. An alternative use for rubrics is to allow students to participate in a scoring exercise. This works as follows Choose a small number (say, 5-7) of student responses (ideally from another year or class). Remove all identifying information. Have the students work in small groups to score the papers. Require the groups to come to consensus about the scores to be assigned and the reasons why. Allow about half an hour for discussion. If you like, have a whole group discussion about the results. This technique works best with students 10 and older. The idea is to help students understand what teachers are looking for in responses.
  • #16 That’s the end of our chapter on rubric development; if you’d like more information, here are some references for you.