ANET Open Response Palooza Presentation DC


Published on

Achievement Network Open Response Scoring Workshop given at Cleveland Elementary School on Tuesday, January 12, 2010

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • 10 min, have share out, list on chart
  • Katie discusses three bullets, then Justin and Lindsay discuss the overarching goal – we frame how we see this pd supporting our schools’ work. We also say that this is NOT a PD about how to improve constructed response scores nor is it a PD about how to review constructed response data during a data meeting (reinforce the objectives).
  • After Katie reviews the content, Justin and Lindsay remind everyone that ANet sends holistic rubrics
  • After Katie discusses content, Lindsay and Justin remind everyone that Anet uses constructed response (not composition)
  • Katie will show first line and then ask if anyone has experienced scoring bias. Take one response about an experience with scoring bias. Katie will define what scoring bias is and then ask for examples of what might cause scoring bias (then bring up the bullet points).
  • After Katie reviews the process, Justin and Lindsay jump in and explain that this is the process you want to take back and replicate in your school. Even though we’re using a holistic rubric, you could use this with composition as well (steps 2-5: you just wouldn’t have a passage to read). These slides would be what you could do in your building if you replicated this PD.
  • Justin and Lindsay might, depending on time and participation, chime in and ask how this is already being used or might be used in buildings.
  • People will show their own score using colored cards.
  • People will show their own score using colored cards.
  • People will show their own score using colored cards.
  • People will show their own score using colored cards.
  • Step 1: Ask them to go through and highlight key words on the rubric before they begin scoring
  • Step 1: Ask them to go through and highlight key words on the rubric before they begin scoring
  • Norm yourself first. Make key points list before you begin grading. Katie Share what we came up with for Part A…allow 5 minutes for teachers in pairs to create part B. Share out.
  • Katie Model – possible that we cut this depending on time
  • Katie model – possible that we cut this, depending on time
  • 20 minutes, Tara, Katie, Lana circulate
  • Justin and Lindsay can reinforce bringing this back to schools to increase validity of constructed response data.
  • ANET Open Response Palooza Presentation DC

    1. 1. ConstructedResponse Palooza<br />Winter 2010<br />
    2. 2. Think, pair, share<br />What are some of the challenges you face <br />when grading constructed response<br />questions? <br />
    3. 3. Objectives<br /><ul><li>Provide teachers with a firm understanding of the different types of rubrics and how to use them with different student writing samples.
    4. 4. Provide an opportunity to examine and discuss an anchor set for a given writing prompt.
    5. 5. Provide teachers with the opportunity to examine student samples from an administered constructed response item and norm along the ANet rubric.</li></ul>OVERARCHING GOAL: To leave with a better understanding of using rubrics to effectively score constructed responses and to take this back to your schools to improve your school’s overall use of constructed response data.<br />
    6. 6. agenda <br />I. Rubrics 101 15 min.<br />Different types of rubrics<br />Different types of questions/prompts<br />Scoring <br />II. Norming around an Anchor Set 45 min.<br />Review of passage, item, and rubric<br />Model of Scoring<br />Paired Scoring and Debrief <br />II. Practice Norming around ANet Responses 1 hour<br />Groups use rubrics to score student responses<br />Debrief<br />IV. Next Steps 10 min. <br />
    7. 7. Rubrics 101Holistic v. Analytic<br />Holistic Rubrics<br /><ul><li>provide a single score based on an overall impression of a student’s performance on a task.</li></ul>DC Uses: to score constructed response items aimed at assessing reading comprehension<br />Advantages: quick scoring, provides overview of student achievement <br />Disadvantages: does not provide detailed information, may be difficult to provide one overall score <br />Analytic Rubrics<br /><ul><li>provide specific feedback along several dimensions.</li></ul>DC Uses : to score writing compositions aimed at assessing writing <br />Advantages: more detailed feedback, scoring more consistent across students and graders <br />Disadvantages: time consuming to score <br />
    8. 8. ELA Composition vs. Constructed Response Questions<br />Composition<br /><ul><li>Assess writing
    9. 9. A starting point to initiate a student’s own thoughts on the topic, supported by their experiences and ideas
    10. 10. No “correct” answer (but must address topic specified by the prompt)
    11. 11. Style and organization are important to the score
    12. 12. Uses Analytic Rubric and scored in two domains: Topic Development and Conventions
    13. 13. Scored out of 10 points in grades 4,7 and 10</li></ul>Constructed response<br /><ul><li>Assess reading comprehension
    14. 14. Student must answer the question asked and answer must be supported with details from the text
    15. 15. Answers found in reading the passage or can be inferred from information in the passage
    16. 16. The explanation and support from the text matter much more than style and organization
    17. 17. Uses Holistic Rubric, conventions are not scored
    18. 18. Worth 9 total points (3 CRs, each worth 3 pts.)</li></li></ul><li>Sample Composition Writing Prompt<br />Think about a time you were helpful. For example, maybe you helped a new student feel comfortable in school, helped a teacher with a job in the classroom, helped with a project in your neighborhood or school, or helped to care for an animal.Write a story about a time you were helpful. Give enough details for readers to understand how you were helpful.<br />Example is from the 2009 4th grade ELA MCAS<br />
    19. 19. Analytic Rubric<br />Topic/Idea Development <br />
    20. 20. Analytic Rubric<br />Standard English Conventions<br />
    21. 21. Sample Constructed Response Question<br />Based on the article, describe what people<br />have learned from studying slave quilts.<br />Support your answer with important<br />information from the article.<br />Example is question 35 from the 2008 6th grade ELA MCAS<br />Passage “Pattern for Freedom: Women’s Quilt as Art”<br />
    22. 22. Holistic Rubric<br />
    23. 23. Scoring Bias in Constructed Response items <br />It’s important for validity not to let writing skills <br />impinge on reading comprehension scores.<br />Things to look past (to avoid scorer bias)<br />Penmanship<br />Neatness<br />Mechanical errors<br />Spelling<br />Length<br />
    24. 24. Process for scoring student responses to Constructed Response Prompts<br />Read the passage and the accompanying prompt.<br />Review the rubric for the prompt and highlight key words.<br />Create a list of scoring notes. <br />Begin to build an anchor set.<br />Score remaining student responses using rubric and anchor set.<br />
    25. 25. Read passage and accompanying prompt<br /><ul><li>Read the passage </li></ul>“Pattern for Freedom: Women’s Quilt as Art”<br /><ul><li>Read the prompt:</li></ul> Based on the article, describe what people<br /> have learned from studying slave quilts.<br /> Support your answer with important<br /> information from the article.<br />
    26. 26. Review the rubric<br />
    27. 27. highlight key words in the rubric<br />
    28. 28. Create Scoring Notes<br />What people have learned from slave quilts:<br /><ul><li>Quilts were forms of self-expression: used to pass on emotion, histories and religious beliefs
    29. 29. Quilts were a way to convey messages in African culture:</li></ul>Historical symbols: such as X for a crossroads, snake motif for the West African God of fertility, flower patterns for the Haitian goddess of love, etc.<br /><ul><li>Quilts were a way for slaves to leave a record of their lives (personal histories):</li></ul> The use of color: for example red for a woman’s birth process or a man’s role as hunter and warrior, etc.<br /><ul><li>Quilts were a way to guide runaway slaves (use in the Underground Railroad):</li></ul>Black fabric symbolized a safe house, the “Drunkard’s Path” told slaves to take an indirect route, etc.<br />
    30. 30. Create an Anchor set<br />What is an Anchor set?<br /><ul><li>An anchor paper is a sample response that illustrates student work at a given score point on a rubric.
    31. 31. An anchor set will include examples of student responses at each score point at the rubric.
    32. 32. These student samples help illustrate what is provided in the rubric.</li></li></ul><li>Student Response #1<br />Score: 0<br />Why a 0?<br />The response is incorrect.<br />Why not a 1?<br />The information contained in the response is incorrect and not supported by the article.<br />
    33. 33. Student Response #2<br />Score: 1<br />Why a 1?<br />There is minimal attempt to explain that slave quilts contain meaningful designs. There are no supporting details from the article included.<br />Why not a 0?<br />The information provided was not incorrect. The response, while lacking details from the passage, does show understanding of quilts helping slaves escape from slavery and that designs had different meanings.<br />
    34. 34. Student Response #3<br />Score: 2<br />Why a 2?<br />The response is a general description of what people have learned from studying slave quilts. The focus of the discussion is on the symbolism of the colors and patterns used in slave quilts.<br />Why not a 3?<br />The response is partially complete in that it lacks any reference to the use of quilts for passing down history. Details from the story are general.<br />
    35. 35. Student Response #4<br />Score: 3<br />Why a 3?<br />The response clearly, completely, and accurately explains that the study of slave quilts has taught people about the personal history of slaves, aspects of African culture, and symbols used in the Underground railroad. Relevant quotes from the article are included to support each idea. The introduction and conclusion do not contribute any additional information to this response.<br />Why not a 2? <br />The essay is clear and complete. Specific details, including quotes from the article, are used in the response. No inaccurate details are included.<br />
    36. 36. Think, Pair, Score<br />Read student samples #5 and #6.<br />Score each sample using the rubric.<br />Justify your score.<br />Share your thoughts with your neighbors.<br />
    37. 37. Student Responses #5 and #6<br />Student Response #5<br />Score: 3<br />Why a 3?<br />The response is a clear, complete and accurate description of what people have learned from studying slave quilts. Details are included to explain the meaning of several historical symbols as well as quilt patterns designed to guide runaway slaves. The symbolic meaning of different colors are also discussed.<br />Student Response #6<br />Score: 2<br />Why a 2?<br />This response provides a partial explanation of what people have learned from studying slave quilts. Information is provided about the symbolism of the color black and the significance of certain patterns, but overall details are limited.<br />
    38. 38. Scoring an ANET prompt<br />Read the passage and the accompanying prompt.<br />Review the rubric for the prompt and highlight key words.<br />Create a list of scoring notes. <br />Begin to build an anchor set.<br />Score remaining student responses using rubric and anchor set.<br />
    39. 39. ANet Writing Prompt<br />Read the passage:<br />Take a moment to read the passage from Lost in the Grand Canyon<br />by Deborah Hopkinson.<br />Read the prompt<br />According to the article, Powell was “a born leader with courage and<br />ambition.” Based on the article do you agree or disagree with this<br />statement? Why? Use evidence from the article to support your<br />answer.<br />
    40. 40. Review the Rubric<br />
    41. 41. Highlight Key words in the rubric<br />
    42. 42. Scoring Notes (things to look for)<br />Powell is a born leader<br />Willing to search last unknown area (Grand Canyon).<br />Fought in the Civil War- lost arm, unable to row, but still leads journey.<br />Shouts orders and warnings when river turns wild.<br />Races downstream searching for the boat.<br />Comes up with plan to ‘run down a chute’.<br />Shoots off rifles and waits for the men who left- before continuing on the journey.<br />Powell and crew make it safely down the Colorado River.<br />Powell is not a born leader<br />Places blame on others when things go wrong.<br />Orders crew members around.<br />Powell is unprepared- doesn’t know how long trip will take and there isn’t enough food.<br />Tries to persuade crew members to stay, but is unsuccessful.<br />
    43. 43. Model Scoring- Response A<br />Please refer to Response A<br />Score: 0<br />Why a 0? <br />All details/information provided in response are incorrect.<br />Why not a 1?<br />Student does not include any relevant and correct details in response to support a position.<br />
    44. 44. Model Scoring- Response B<br />Please refer to Response B<br />Score 2: <br />Why a 2?<br />Student provides a partial explanation of why Powell was a born leader. Details are included about Powell attempting to save men after the boat went over the waterfall and that he did not allow his men to starve. The overall details are limited and the response is not complete. <br />Why not a 3? <br />The response includes some relevant details from the passage to support the position, however response is not complete.<br />
    45. 45. Begin to Build an Anchor set<br />You will be receiving a set of papers for this<br />prompt. <br />Unfortunately, each scoring point on the rubric does not have a student sample.<br />Independently and then in pairs take some time to<br />review each of the papers in light of the rubric<br />and provide a score for each one. <br />Please make notes for each example as to why you<br />believe it deserves that score.<br />
    46. 46. Anchor Set<br />Response C<br />Score: 2<br />Most examples contained in response were relevant and supported the position of Powell being a born leader. Response was partially complete.<br />Response D<br />Score: 1<br />Response is a minimal description of Powell being a born leader. Supporting details from article are limited and there is little attempt to explain how or why Powell should be viewed as a born leader.<br />Response E<br />Score: 1<br />This response includes a minimal description of Powell being a born leader. Some details are incorrect or irrelevant. <br />
    47. 47. Anchor Set<br />Response F<br />Score: 0<br />Response only restates the prompt. No information from passage is included. Response demonstrates no description of whether Powell is a born leader.<br />Response G<br />Score: 2<br />Response includes multiple relevant details to support Powell being a born leader. Specific details were included to support thoughts presented. Response was not complete.<br />Response H<br />Score: 2<br />This response includes relevant details. Final section seems to include inaccuracies and is incomplete.<br />
    48. 48. Takeaways<br />
    49. 49. Best practices<br />
    50. 50. Useful Resource for Constructed Response<br /><ul><li>Find sample prompts and student responses on the Massachusetts Dept. of Elem. and Secondary Ed. website: Link to MA DOE Released CRs</li></li></ul><li>Next Steps<br /><ul><li>Feel free to exchange contact information with the individuals in your scoring group to keep in contact about constructed response items and scoring.
    51. 51. Please take the anchor set, passages, rubrics and student responses with you.
    52. 52. Please contact us with any questions you might have.
    53. 53. Please bring this back to your school so that over time you can collect increasingly valid constructed response data.</li></li></ul><li>Thank you so much for coming.We want your feedback!<br />