Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Seven purposes presentation

206 views

Published on

John Cronin's presentation: Multiple Measures Done Right - The Purpose Driven Assessment System at the January, 2017 COSA conference

Published in: Education
  • Login to see the comments

Seven purposes presentation

  1. 1. Multiple measures done right: The purpose- driven assessment system John Cronin, Ph.D. VP – Education Research Northwest Evaluation Association
  2. 2. The pursuit of compliance is exhausting because it is always a moving target. Governors move on, the party in power gets replaced, a new president is elected, and all want to put their own stamp on education. It is saner and less exhausting to define your own course and align compliance requirements to that. The problem of compliance-based assessment systems
  3. 3. Seven standards that define the purpose driven assessment system
  4. 4. The purposes of all assessments are defined and the assessments are valid and useful for their purposes. DEFINE YOUR PURPOSES
  5. 5. Northwest Evaluation Association and Gallup - 2016 https://www.nwea.org/content/uploads /2016/05/Make_Assessment_Work_for_ All_Students_2016.pdf Make Assessment Work for All Students: Multiple Measures Matter
  6. 6. TEACHERS, PRINCIPALS AND SUPERINTENDENTS say students spend too much time taking assessments.
  7. 7. Percent of students who say the results from state accountability tests are useful 29 Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter. https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf
  8. 8. Teacher and Administrator perceptions on the purposes of assessments 0% 10% 20% 30% 40% 50% 60% 70% 80% State Policymakers Parents Students Teachers Administrators % State accountability assessments % Classroom tests and quizzes Principals: Do you believe the purpose of data from (state accountability assessments, classroom assessments and quizzes) in your district is well understood by each of the following:
  9. 9. Students find many assessment types useful when the assessment is tied to a specific purpose Classroom tests and quizzes developed by teachers to evaluate their students’ learning process Interim assessments that are given to students two to three times over the year so teachers can see where students are growing and where they need to learn more Performance tasks that require students to apply things they have learned to a real-world problem or situation Formative assessments 78% 76% 75% 74%
  10. 10. One action to take First, document and prioritize your purposes for assessment then… inventory the assessments in use and evaluate their alignment with your purposes. It’s a big job but worth the time!
  11. 11. Teachers are educated in the proper administration and application of the assessments used in their classrooms. EDUCATE TEACHERS
  12. 12. What types of extra help, if any, does your teacher or school give you if you do not do well on a… CLASSROOM TEST STATE STANDARDIZED TEST Before and/or after school support 43% 24% One to one support 32% 19% Access to added resources 29% 22% Small group support 24% 17% No extra support provided 20% 38%
  13. 13. Teachers need to … Understand the purpose Follow the procedures Properly interpret and act
  14. 14. One action to take Develop internal structures and policies to provide training for all teachers (and administrators) in your assessments and promote proper use. This training should help educators understand what the results can and can’t tell them, and how best to use it to make a difference for students.
  15. 15. Assessment results are aligned to the needs of their audiences. ALIGN RESULTS
  16. 16. Assessment audiences • School Board • Students • Teachers • Parents • Principals, school administrators, and teacher leaders • District Administrators • Community members • State and federal officials
  17. 17. Percentage of parents who say teachers rarely or never discuss their child’s assessment results with them 61 Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter. https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf
  18. 18. Questions parents want answered from assessment • Core question - Do you know and care for my child? • What are my child’s strengths and weaknesses? • Is my child on track for the next grade level? • Is my child on track for college? • Is my child showing improvement? • Should I be concerned?
  19. 19. What kind of data do parents want? 95% 95% 93% 92% 91% 90% 88% 84% 79% 77% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Monitoring general progress Knowing when to be concerned Determining readiness for next grade Knowing if I need to seek extra help Monitoring standards Communicating with teacher/admin Measuring high quality teaching Helping my child with homework Adjust content to student needs Providing activities for home Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC
  20. 20. Questions teachers want answered from assessment • What does each student know and not know? • What does this student need to learn next? • What resources will help this student? • How can I group these students for instruction?
  21. 21. One action to take Train teachers in the skills needed to have a positive and productive parent conversation. Many teachers find these conversations intimidating. When educators are well prepared they gain confidence and develop higher levels of trust and support from most parents.
  22. 22. Redundant, mis-aligned, or unused assessments are eliminated. STREAMLINE MEASURES
  23. 23. Your current assessment tools •What purposes do each assessment meet well? • Comes from assessment design and capabilities – not your current use • Is there alignment between the assessment and the information needed? •How are you using the assessments? • Reduce use to within the design and capability • Use assessments more fully to meet other purposes
  24. 24. One action to take Identify the places where you have multiple assessments in-use validly performing the same purpose. Determine whether you can leverage other assessments to meet these needs.
  25. 25. Assessment results are delivered in a timely and useful manner. DELIVER RESULTS
  26. 26. Percent of students who say they do not receive their state accountability test results. 37%
  27. 27. For parents, assessment results begin losing their relevance within… 1month after the assessments are administered Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC https://www.nwea.org/content/uploads/2014/07/NWEAGRUNWALD_Assessment_Perceptions_b.pdf
  28. 28. Timely results are related to use If needed for instruction, results need to be faster than school improvement conversations Reports designed to fulfill purpose • School administrators want reports that quickly identify students who need help. • Teachers want reports that help them make instructional decisions. • Parents want to know whether there student is on track and to be warned when trouble is ahead.
  29. 29. One action to take Review your assessment calendar, with a focus on when data is being delivered to your audiences. Adjust your testing calendar or practices to ensure that data gets to educators when they most need it. Make sure assessment results are timed to fit your schedule for professional days to encourage dialogue.
  30. 30. The metrics and incentives used encourage a focus on all learners. SUPPORT ALL LEARNERS
  31. 31. Grade Students Athletes % Fine Arts % 9 3276 1487 50% 2256 76% 10 3190 1215 41% 1656 56% 11 2967 932 31% 1233 42% 12 2795 703 24% 1028 35% Percent of students participating in athletics and fine arts programs
  32. 32. What gets measured and attended to really does matter 0 5 10 15 20 25 30 35 40 165 167 173 176 181 185 187 189 191 194 196 198 200 202 204 206 208 210 212 214 216 218 220 222 224 226 228 230 232 234 236 238 240 242 244 246 248 250 252 254 257 262 NumberofStudents Fall RIT Mathematics No Change Down Up Proficiency College Readiness One district’s change in 5th grade mathematics performance relative to the KY proficiency cut scores
  33. 33. Changing from Proficiency to Growth means all kids matter 0 5 10 15 20 25 30 166 169 175 178 183 186 188 190 193 195 197 199 201 203 205 207 209 211 213 215 217 219 221 223 225 227 229 231 233 235 237 239 241 243 245 247 249 251 253 255 258 266 NumberofStudents Student’s score in fall Mathematics Below projected growth Met or above projected growth Number of 5th grade students meeting projected mathematics growth in the same district
  34. 34. One action to take Ask the question – “What 3 to 5 metrics drive educational decision making in our school or district? Now ask – “What behavior do those metrics incentivize and are all students encompassed in those metrics? Make changes as needed to be sure your metrics encourage focus on all students.
  35. 35. The assessment program contributes to a climate of transparency and objectivity with a long-term focus INVITE COLLABORATION
  36. 36. “Everyone is entitled to their opinion, but not to their own facts.” Daniel Patrick Moynihan
  37. 37. What is accountability? Accountability is a dialogue between the stakeholders and the leaders of their schools. Part of that dialogue is understanding and aligning the goals and objectives of the parents, the schools, and the larger community. Another part of that dialogue is discussing how your schools are doing in reaching those goals. The most important part of the dialogue is what you’re doing to improve performance based on this information, which is leading.
  38. 38. The four principles that guide your communications to your audiences. • Clearly identify who your audience is and address them as your partner. • Understand what that audience wants to know about your schools and their performance. • Be transparent about how you are performing on these objectives. • Communicate your strategy to improve performance, and corrective action when strategy fails.
  39. 39. Florida District Highly Effective Effective Needs Improvement Developing Unsatisfactory VA Score Florida Ranking 1 44.4% 55.6% 0.0% 0.0% 0.0% 0.39 109 2 25.0% 75.0% 0.0% 0.0% 0.0% 0.37 121 3 90.9% 9.1% 0.0% 0.0% 0.0% -0.14 2802 4 60.7% 39.3% 0.0% 0.0% 0.0% -0.14 2797 5 81.2% 18.8% 0.0% 0.0% 0.0% -0.16 2831 6 37.3% 54.2% 1.7% 0.0% 6.8% 0.12 880 7 81.3% 18.8% 0.0% 0.0% 0.0% 0.22 402 8 41.7% 55.6% 1.4% 1.4% 0.0% -0.34 3274 9 52.2% 47.8% 0.0% 0.0% 0.0% 0.16 664 10 27.0% 66.2% 1.4% 0.0% 5.4% 0 1764 11 7.1% 72.6% 9.5% 10.7% 0.0% -0.08 2445 Teacher Evaluation Ratings in Eleven Florida Schools 2013
  40. 40. Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June 16, 2016 from http://www.newyorker.com/magazine/2014/07/21/wrong-answer “After more than two thousand interviews, the investigators concluded that forty-four schools had cheated and that a “culture of fear, intimidation and retaliation has infested the district, allowing cheating—at all levels—to go unchecked for years.” They wrote that data had been “used as an abusive and cruel weapon to embarrass and punish.” Several teachers had been told that they had a choice: either make targets or be placed on a Performance Development Plan, which was often a precursor to termination. At one elementary school, during a faculty meeting, a principal forced a teacher whose students had tested poorly to crawl under the table.”
  41. 41. Differences in fall-spring test durations An illustration of gaming 15% 25% 60% Mathematics Spring < Fall Spring = Fall Spring > Fall 0.0 1.0 2.0 3.0 4.0 5.0 6.0 Spring < Fall Spring = Fall Spring > Fall GrowthIndex Mathematics Differences in growth index score based on fall-spring test durations
  42. 42. One action to take Review the tone of a few of your recent communications to stakeholders about student learning. As you read these does it feel like your touting, or are you truly educating your partners? Try writing an op-ed about your educational program that would surprise readers because of its honesty and frankness.
  43. 43. A purposeful assessment program assessment system ensures you have direction, that your direction stays true, and that you and your students eventually reach their intended destination
  44. 44. Thank you for joining us! http://info.nwea.org/MMDR-Guide-LP.html?utm_medium=press- release&utm_source=rss&utm_campaign=multiple- measures&utm_content=text-link Multiple Measures Done Right – can be downloaded from

×