Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to TNB Roundtable slide deck by Mary-Kim Arnold of Rhode Island Foundation
Similar to TNB Roundtable slide deck by Mary-Kim Arnold of Rhode Island Foundation (20)
Recently uploaded
Recently uploaded (20)
TNB Roundtable slide deck by Mary-Kim Arnold of Rhode Island Foundation
- 1. TNB Roundtable: Insights from a Community Foundation’s Evaluation Director Mary-Kim Arnold Director, Evaluation & Learning January 2016
- 2. Agenda • About the Rhode Island Foundation • About me and my role • Your questions • Key concepts • Framework for systems work • Questions / Contact
- 3. About the Rhode Island Foundation • Founded in 1916 – centennial • 800 million in assets – 35 million in grants (including donor advised – “discretionary” is closer to 9 million) • serve the state of RI – largest funder in RI • www.rifoundation.org
- 4. Key concepts • Levels of accountability • Units of analysis • Ratios – meaningful context • Performance vs. impact
- 6. Mountain of Accountability • Basic accountability for management • Accountability for “impact” • Accountability for learning, development, and adaptation
- 7. Units of analysis: who or what? • Individuals • Groups • Geographical units (neighborhood, town, state) • Social interactions (relationships, social behaviors – e.g., getting married or divorced)
- 8. Ratios: in relation to what*? “We served 375 students this year in our after-school program.” * in relation to what unstated condition, assumption, goal, or expectation?
- 9. Meaningful context -- examples • Of the 375 students who participated in our after- school program, 98% enrolled for the following year. behavior • Of the 400 eligible students in Central Falls, we served 375 in our after-school program. reach • Our goal for the year was 200 students, which we surpassed by enrolling 175 students above our goal. implementation or demand
- 10. Meaningful context + unit of analysis • student: Of the 375 students who participated in our after-school program, 98% enrolled for the following year. behavior • teacher: Teachers in our after-school program had 98% retention rate for students between years 1 and 2. performance • organization: This past year, XYZ reached its 5-year goal of 98% retention rate for students. implementation
- 11. Performance vs. “impact” ▷ High-performing organizations vs. high impact organizations ▷ Managing for performance but measuring for impact ▷ How might your role and tasks change if you job was measured for impact?
- 12. High performance vs. high impact* High-performing organization High-impact organization • fiscally strong • good governance • visible, respected leadership • highly-qualified staff • clean audit • growing donor base • transforming lives • ending poverty • closing the achievement gap • ending hep-C * Yes! Certainly, a case can be made that only high-performing organizations can produce results, but the fact that the case can be made does not eliminate the need to distinguish between the two. **
- 13. Systems Change – Areas of Focus Context Components ConnectionsInfrastructure Scale Source: www.ccitoolsforfeds.org/doc/AFrameworkForEvaluatingSystemsInitiative.pdf
- 14. Defining focus areas of systems work •Improving the political environment that surrounds a system so it produces the policy and funding changes needed to create and sustain it. CONTEXT •Establishing high-performing programs and services within the system that produce results for system beneficiaries.COMPONENTS •Creating strong and effective linkages across system components that further improve results for system beneficiaries.CONNECTIONS •Developing the support systems need to function effectively and efficiently.INFRASTRUCTURE •Ensuring a comprehensive system is available to as many people as possible to produce broad and inclusive results for system beneficiaries. SCALE
- 15. Theory of Change menu for systems work Context Components Connections Infrastructure Scale political will – public education campaigns, awareness-building, media advocacy, community mobilization, coalition- building programs – quality programs, services, and interventions network – linkages between and among programs, organizations, lifecycle, coordination capacity – of individual entities or of the network to function effectively access – ability to increase services and numbers served while maintaining system integrity • increases in public awareness • public engagement or mobilization • reframing issue • public will • political will • new or more diversified funding or investments • policy changes • new programs or services developed • expanded individual programs or coverage • improved program quality • increased program efficiency • outcomes that precede impact • coordinated assessments, intake, outreach • referrals across services • coordinated planning across components • shared data or administration • MOUs between system components • entities that oversee and coordinate subsystems • funding streams that are less categorical • leveraged funds • systems-level standards • professional development structures • practitioner training and technical assistance • system spread • system depth • system sustainability • shifts in system ownership DefinitionsAnticipatedOutcomes
- 16. Evaluation design menu for systems work •Has the intervention changed the political environment? •Has the intervention prompted the availability of new investments, policies, or practices that will enable changes in the system? CONTEXT •Did program activities produce the intended benefits? •Were interventions implemented as designed?COMPONENTS •Did the initiative design and implement connections and linkages as intended? •Did the connections and linkages produce their intended impacts?CONNECTIONS •Did the initiative establish infrastructure or supports that are consistent with its objectives? •Did the infrastructure or supports achieve their objectives for effectiveness, sustainability, and quality? INFRASTRUCTURE •Did the initiative enable system scale up with quality and fidelity? •Did scale-up result in broad impacts for beneficiaries at a system-wide population level?SCALE
- 17. Evaluation methodologies menu for systems work CONTEXT Theory of change evaluation Case studies Public polling Key informant interviews or surveys Media tracking and analysis COMPONENTS Program evaluation methodologies Program monitoring Quality assessments Efficiency analysis Customer surveys CONNECTIONS Program evaluation methodologies System mapping Network analysis Customer surveys INFRASTRUCTURE Theory of change evaluation Case studies Performance audits Management information systems Practitioner data collection SCALE Population- based demographic analysis Program evaluation methodologies System or program monitoring Results-based accountability
Editor's Notes
- Blandin Foundation – Mountain of Accountability
- The important part of this is not so much that we have to focus on one or another one of these elements in our portfolios, but rather these are what help us get to how we should evaluate – what we should be looking for. So, in initiatives where we are looking at context – e.g., Backyard – we are looking for changes in the political environment – shifts in investment, policy or practice that will enable changes at other levels – so theory of change is that
- These are possible methodologies – not necessary to use all or even any listed – a menu of options for methodologies that align for each focus area.