Ball Foundation-RUSD Partnership Final Evaluation Report

2,039 views

Published on

The final report (presented to the foundation board) of an evaluation of the Ball Foundation's partnership with Rowland Unified School District, prepared by Catherine Awsumb Nelson, Ph.D., independent evaluator.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Ball Foundation-RUSD Partnership Final Evaluation Report

  1. 1. The Ball-RUSD Partnership Final Evaluation Report September 23, 2011 Catherine Awsumb Nelson, Ph.D.1
  2. 2. Key QuestionsI. What did the partnership look like? a) Design b) Participation c) QualityII. What impacts did it have? a) Non-participants b) Personal transfer of participants c) Broader impactsIII. What does Ball leave behind?IV. What is the potential for sustainability?V. What was learned about investing in district transformation? 2
  3. 3. 2010-11 Data CollectionDATA SOURCE n DetailsIndividual interviews 22 Executive Cabinet (5), SIL design team (2), CoP2 (2), CoPs (7), IC (6)SIL case studies 5 schools At each of 5 schools, fall and spring interviews with principal, 30 new team member, and returning team member, plus interviews document analysisEvent observations 9 SIL (3), Literacy Network (3), IC (1), Efficacious Instruction workgroup (1), Sensing team (1)Event feedback 10 SIL (5), Literacy Network (4), IC (1)surveysMilestone 8 Design teams from LN, SIL, and IC, full Executivemonitoring meetings Cabinet Middle and end of yearStaff survey 364 131 partnership participants (58% response) 233 non-participants (35% response)Student survey 4949 Overall response rate 56% for Grades 4-12Participant tracker 3 895 Certificated staff only
  4. 4. Ball approach to district transformation Transform, not reform Focus is on systems change Assumption that most necessary expertise is already in the district Key strategies:  Capacity building  Coherence making 4
  5. 5. Why is “coherence” so important? “I hope we are strong enough  “The „here‟s another binder‟ to keep it going. I worry about mentality is what we fight here it. Not a reflection on the work all the time. So many things Ball has done but the they want you to know and district…we do amazing things learn about. We dip our toes in this district, but we tend to into so many things and they all dabble. You have to keep blur together. I don‟t think we things alive yourself if they are need to try so many things at working for you because the once. With kids, when you district moves on. We do a lot bombard them with stuff, it of good things but don‟t stick doesn‟t work. When you teach with anything long enough to for depth, spend the quality make it great.” (CoP member) time on a unit, that is when learning happens. (SIL member) Fewer than 10% of respondents agreed with the statement “Once we start a new program in this district we follow up to make sure it is working.” 5
  6. 6. Critical Features of the RowlandPartnership Ball provided structures and time for inquiry and collaboration around instruction at three levels  Classroom  School  District Collaboration supported through design and coaching Focus on changing adult learning to change student learning Working with the willing 6
  7. 7. Flipping the adageThe old adage is time is money.In education, money buys time, which creates opportunity for learning. That isthe great gift Ball has given Rowland, along with design and coaching that helpthe professional time align with what we know about what creates impact in theclassroom. 7
  8. 8. U.S. Schools Lag International Competitorsin Providing Professional Learning Time The United States is far behind in providing public school teachers with the kind of high intensity, job-embedded collaborative learning that research shows is most effective in changing practice and improving learning U.S. teachers report little professional collaboration in designing curriculum and sharing practices, and the collaboration that occurs tends to be weak and not focused on strengthening teaching and learning. Compared to other nations that outperform the United States on international assessments, American teachers spend much more time teaching students and have significantly less time to plan and learn together. “Professional Learning in the Learning Profession: A Status Report on Teacher Development in the U.S. and Abroad.” (National Staff Development Council, 2009) 8
  9. 9. Building Instructional Capacity in RUSDStructure Function Ball supports #Communitie Small groups of educators •$ for professional books 120-s of Practice collaborating around a and training 150 specific literacy topic •Literacy Network Days •Cluster days •CoP Garden •Developmental frameworkSchool Teams of 6-8 •Design and facilitation for 110-Instructional administrators and monthly cross-district 150Leaders teachers tasked with meetings leading professional •Learning Walks learning at each siteInstructional Representatives across •Meeting design and 25+Cabinet district and role groups facilitation charged with identifying •Support for workgroups and supporting district wide instructional priorities 9Executive Existing district leadership •Coaching 5
  10. 10. District Context Budget cuts Accountability pressure Demographic shifts Hollowing out of district instructional support capacity 10
  11. 11. Partnership Participation,2009-2011 Both years, 16.8% 2010 only, 8.9% Never, 65.9% 2011 only, 8.4%11
  12. 12. Intermediate and secondary participation increased40% 35%35% % of classified employees participating in 33% 2010 201130% 28%25% Partnership 21%20% 19%15% 14%10%5%0% Elementary Intermediate High 12
  13. 13. Quality of the Work13
  14. 14. Ratings for Quality of Professional Learningremarkably stable% of participants rating partnership professional 2011 2010learning as “Good “ of “Excellent”Ensuring that all voices are heard 86% 86%Having a positive impact on student learning 78% 77%Making it safe to raise difficult issues 78% 74%Building on existing professional expertise within thedistrict 78% 76%Focusing on issues directly relevant to my practice 76% 75%Being grounded in data and/or other evidence ofstudent learning 71% 67%Striking a good balance between content and process 69% 71%Using time well 57% 58% 14
  15. 15. Measures of overall intervention quality increased across the board, with the biggest increase in potential for sustainability Ball Partnership work has helped RUSD move 57% different parts of the system toward a common 71% focus. District leaders have demonstrated that they are 61% committed to the Partnership work. 61% Ball Partnership work is well integrated into the 33% 2010 day-to-day work of the district. 45% ParticipantsBall Partnership activities help us deeply examine 69% our approach to teaching and learning. 78% Working with Ball helped RUSD respond more 49% strategically to Program Improvement status and 59% budget cuts than we otherwise would have. Important aspects of the work will continue once 42% Ball personnel and funds are no longer in the 65% district. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % agree/strongly agree 15
  16. 16. Impacts on non-participants16
  17. 17. All avenues for non-participant awareness increased thisyear, with informal communication about CoP work growing themost Heard about Ball Partnership activity in a 46% staff meeting 56% A colleague talked to me about work he/she 42% was doing with Ball 47% I heard informally about the work a teacher 22% Community of Practice was doing 39% Participated in a meeting led by my schools 31% School Instructional Leadership team 36% Read about Ball Partnership activity on the 24% district website or in printed materials 33%Participated in a Learning Walk @ my site led 25% by my schools School Instructional… 29% I was told about the work of the Instructional 13% Cabinet 21% 2010 2011 Work from a teacher Community of Practice 18% group was shared in a staff development… 20% I havent heard anything about the 10% Partnership this year 6% 0% 10% 20% 30% 40% 50% 60% 17 % of non-participants who heard about partnership through…
  18. 18. 40-50% of non-participants noticed improvements in site meetings 100%% of non-participants saying site meetings over the last 90% 80% Much more so More so 70% two years have changed 60% 50% 14% 15% 20% 40% 14% 12% 12% 13% 30% 20% 35% 38% 40% 36% 29% 30% 32% 10% 0% Driven by Inquiry Uses Build Build cultural Focused on Opportunities what based evidence of professional proficiency teaching and for teachers student learning learning collaboration want/need to learning community learn 18
  19. 19. Participant Impacts19
  20. 20. Reported levels of personal transfer areunchanged from last year 100% Revolutionized my practice Major transfer Moderate transfer% of participants reporting level of transfer 90% 80% 70% 36% 34% 60% 40% 38% 33% 50% 38% 40% 30% 37% 37% 34% 31% 32% 20% 26% 10% 9% 9% 9% 9% 9% 7% 0% My classroom How I prepare How I support How I work How I structure How I connect instruction for and reflect classroom with other staff adult learning my work to on my instruction broader district classroom priorities instruction 20
  21. 21. % of participants rating impact major/moderate 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%21 Collaboration in 43% 43% professional learning Quality of professional 41% 40% learning CoPs as a model of 44% 38% collaborative inquiry Norms of reflective practice/rigorous 41% dialogue 34% Opportunities for teacher 46% 33% leadership Coherent vision of 48% 32% effective instruction Use of effective/research-based 49% 30% practices for instruction SIL teams developing 41% 26% site-level coherence Improved decision- making processes in the 34% 20% district Major District better structured Moderate to support effective Participant ratings of broader district impacts 47% 16% instruction IC developing district- level coherence in 44% 14% instruction
  22. 22. For all impact areas with a direct comparison fromlast year, ratings of Major impact increased 2011- Participants rating impact Increase from "Major" 2010Increased collaboration in professional learning 43% +17%Increased quality of professional learning 40% +12%Establishing norms of reflective practice and rigorousdialogue about instruction 34% +17%Use of effective/research-based practices forinstruction 30% +13%Improved decision-making processes in the district 20% +10%District better structured to support effective instruction 22 16% +6%
  23. 23. District instructional support capacity Staff are held accountable for realizing the 39% districts vision of quality instruction District priorities are clearly focused on 35% supporting and improving instruction The district has a coherent vision of quality 34% instruction District decisions are grounded in data 30% Key resources of time, money, and personnel 28% are clearly connected to instructional… It is clear where and how decisions about 20% instruction get madeDistrict-level decisions are made with adequate 13% input from school-based personnel 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % of participants who agree/strongly agree 23
  24. 24. Capacity Building Structures24
  25. 25. CoPs took off in 2nd half of year 100% 96% 94% 93% 90% 89% 88%% of members saying their CoP had done "Mostly" or 82% 81% 80% 76% 70% 71% 69% 67% 60% Used a cycle of plan/act/reflect 58% 50% "Completely" 43% Jointly examined artifacts of student 40% learning 36% 30% Put in place ways to capture our learning 20% 18% Shared learning with colleagues not 10% in our CoP 0% January March June25
  26. 26. Communities of Practice participant ratings of rigor up sharply over last yearWe routinely used Process Learning Circles and/or 2011 66% specific conversational processes like ordered 2010 59% sharing All of our members stayed engaged and 77% accountable to each other 63% We pushed each other to be rigorous about what 81% works and why in the literacy practice we were 57% focusing on We routinely looked at evidence of student work 81% from our own classrooms as we talked about how 54% well a specific practice worked We routinely agreed to try specific things in our 90% classroom and then discuss with the group how 76% they worked My CoP had a clear question or purpose to focus 90% our inquiry 84% 0% 10% %20%CoP members agree/strongly agree 90% 100% of 30% 40% 50% 60% 70% 80% 26
  27. 27. Greatest SIL accomplishments were in team development;whole staff learning also significantly impacted Learned to function well as a team 48% 43% Built our own competencies as instructional 23% 63% leaders Made staff meeting time more focused on 36% 42% learning Made staff meeting time more collaborative and 29% 48% inquiry based Built shared understanding among our team 16% 57% Completely about what efficacious instruction looks like MostlyBecome seen by all staff as leaders of learning in 19% 48% the school Used cultural proficiency as a lens to analyze 13% 49% instruction in our schoolBuilt shared understanding among the whole staff 12% 44% about what efficacious instruction looks likeUsed learning walks to reflect on instruction in our 27% 29% school 27 0% of members rating extent to which their70% accomplished goal (May SI % 10% 20% 30% 40% 50% 60% team 80% 90% 100%
  28. 28. SIL impact ratings up across the board thisyear Change from last% of SIL participants who agree/strongly agree 2011 yearBecause of the SIL work, people across the district are starting to usemore similar language about instruction 65% 26%The SIL work has significantly influenced our site-level professionaldevelopment approach and agenda 57% 11%Our SIL team had enough representation to effect change in our site 49% 17%Expectations for implementing the SIL work at our site were clear 43% -3%Our SIL team will be a driving force in our schools instructionalimprovement efforts going forward 56% 11%SIL has given teachers more of a leadership role over instruction inthis school 51% 8%We made progress this year in making instruction more public in thisschool 59% 11% 28
  29. 29. Instructional Capacity assessments of theirown effectiveness vary widely across goals Launching a system wide pilot of the new data… 95%Building our own understanding of RTI (within IC) 91%Determining training needed for new data system 74% Setting a direction for RTI district wide 67% Setting district wide instructional priorities 63% Understanding ICs role 55% Monitoring the implementation of the PI plan 35% Supporting staff in implementing instructional… 31% Increasing district wide awareness of IC 23% Monitoring the effectiveness of PI plan… 16% 0% 20% 40% 60% 80% 100% % of IC rating group group “Effective” or “Extremely Effective” 29
  30. 30. Challenges to district-level coherence-making:Imbalance between mandate and resources Capacity Mandate: Build instructional coherence through developing and Lines of communication supporting Authority instructional priorities30
  31. 31. Summary of Impacts: What does Ball leave behind?31
  32. 32. Areas of most and least change this yearGreatest progress Least progress Rigor/depth of learning  Cross strand for partnership connections participants  New structure for district Quality of professional coherence continued to learning for ALL district staff struggle Concrete agreements  Staff assessment of around Efficacious district instructional Instruction capacity District  Levels of personal ownership, confidence transfer stayed flat in sustainability 32
  33. 33. Summary of Impacts Idea of design Expectation that decisions will be collaborative and inclusive Transitions with Expectation that professional learning momentum will be collaborative CoPs as a vehicle for teacher directed inquiry into practice Learning Walks starting to de-privatize practice in some sites SIL starting to re-shape site-based learning 33
  34. 34. “No going back” “One of the most striking places I saw the impact was when we had this presenter from county on EL issues. The way they presented was just so foreign from how we do things- it showed how far we have come. It was just, throw up a power point and then we will take your questions- boom. Instead of taking a piece and really working it the way we do now. (Principal) “I think many of us have passed the point of no return this year. We don‟t want to go back. There is no way we are going back to professional development that is not collaborative and self- initiated.” (Teacher) 34
  35. 35. Summary of Haven or silo? Impacts Struggle to balance accountability with Challenges to the emerging “learning as a journey” Divergent conceptions of assessment Traditional conceptions of teacher system “leadership” Search for plug-in solution still evident in some areas 35
  36. 36. What does Ball leave behind? Morale maintained during difficult time Cuts made with more intentionality Capacity for the design and facilitation of adult learning (in a much broader base of staff) District owns new structures for learning and leadership Norms about adult learning Agreements about efficacious instruction 36
  37. 37. Looking Forward: Sustainability Potential37
  38. 38. Majority of participants are optimistic thatmost impacts will be lasting Collaboration in professional learning 65%Use of effective/research-based practices for… 62%Norms of reflective practice/rigorous dialogue 59% Coherent vision of effective instruction 59% Quality of professional learning 58% Opportunities for teacher leadership 56% CoPs as a model of collaborative inquiry 54% SIL teams developing site-level coherence 53% IC developing district-level coherence in… 44% District better structured to support effective… 41% Improved decision-making processes in the… 33% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % of participants saying impact will "Definitely" or "Probably" last 38
  39. 39. Large increase in confidence that impactsare sustainable Change in Sustainability RatingCommunities of Practice as a model of collaborativeprofessional inquiry +36%Increased collaboration in professional learning +35%School Instructional Leadership (SIL) teams as a means todevelop site-level coherence in instruction +34%Increased quality of professional learning +31%Improved decision-making processes in the district +14%The Instructional Cabinet as a means to develop district-level coherence in instruction -1% 39
  40. 40. Moving Forward in RUSD: Efficacious Instruction40
  41. 41. RUSD Learning STUDENTS: Paradigm principles Learner centered ADULTS: Teaching for Linked paradigm shifts in Learning for Effective adult and student Effective Learning: learning Teaching Democratic Both grounded in brain- Relationships mind principles Inquiry Clarity Collaboration Process of strands is Invested Data Cognition Learning for Effective Ownership Feedback Teaching Expert learners Content for strands is Teaching for Effective Learning41
  42. 42. Process for creating and enacting theframework embodies “capacity” Bottom up Incorporated research and practitioner knowledge Back and forth between the strands “Not a thing” Ongoing opportunities for meaning-making vs. “Rollout” 43
  43. 43. Baseline findings about EfficaciousInstruction in RUSD Teacher clarity is the strongest domain, relationships and engagement are the weakest No significant differences in how Hispanic students experience instruction Quality of instruction as experienced by students drops slowly from 4th to 8th grade, bottoms out in 9th, then climbs again until 12th Compared to students, teachers overestimate the quality of relationships, underestimate quality of feedback 44
  44. 44. What was learned about districttransformation? Slower than reform People want to “thingify” Monitor the balance between ownership and coherence Changing power relationships at the top is hardest  Broadening the teacher role isn‟t easy either People do need to be taught skills of collaboration and inquiry Capacity and buy-in are easier to build in the process of doing authentic work 45

×