Evaluating community Evaluating collaborationEvaluating capacity<br />Tom Kelly Annie E. Casey Foundation   <br />Tanja Ku...
Annie E. Casey Foundation<br />Established in 1948 by Jim Casey, founder of UPS <br />Casey “cousins” with Casey Family Pr...
Annie E. Casey Foundation<br />Began grantmaking in 1980’s<br />State child welfare reform<br />Juvenile detention reform<...
Overview of Comprehensive Community Initiatives (CCIs)<br />Goals focus on multiple levels, including individual/family, n...
Challenges in Evaluating Comprehensive Community Initiatives:<br />Outcomes are either too distal (reduced poverty rates) ...
Range of Tools for a Continuum of Data Needs/Questions<br />Outcome Evaluation<br />Surveys<br />Anecdote<br />Observation...
Measuring Success<br />Change in PLACE<br />Change in PEOPLE<br />Change in COMMUNITYCAPACITIES<br />Change in SYSTEMS OF ...
RBA Model (in one slide)<br />Who are we concerned about?  Population<br />What conditions or change do we want for this p...
Results…what difference did we make in terms of:<br />IMPACT on children and families<br />INFLUENCE on policies, practice...
 Impact, Influence, Leverage<br />We seek to make a positive difference in the lives of children and families through thre...
Place-based community accountability<br />Program-level AND population-level  (whole neighborhood) accountability<br />Bey...
Accountability requires…<br />Funders and decisionmakers (i.e., those with power and resources) are held accountable for t...
Results Accountability<br />www.resultsaccountability.com<br />http://raguide.org/<br />www.aecf.org   “Casey ResultsNet”<...
Weed & Seed<br />Collaboration<br />Coordination<br />Community participation<br />Leverage<br />
Weed & Seed<br />Collaboration<br />Coordination<br />Community participation<br />AUTHENTIC DEMAND<br />Leverage<br />COR...
Measuring Leverage<br />A Practical Guide to Documenting Influence and Leverage In Making Connections Communities<br />Rei...
Making Connections<br />Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based s...
Results/Outcomes<br />Adults have work and earnings<br />Families save, build and protect assets<br />Families access qual...
Local Learning Partnerships (LLPs)<br />Established to support local Making Connections sites (for 10-years and beyond) wi...
Community capacities<br />Program implementation and management<br />Data, self-evaluation, and accountability<br />Commun...
Community capacities<br />SCOPE – coordinated and integrated services and service systems<br />SCALE – policy and system c...
Measuring Community Capacities<br />Began with stages of change framework (INSITES, Bev Parson)<br />
Building a scale<br />Developed common (general) indicators across sites<br />Worked with sites to revise and better defin...
Using the scale<br />Implemented collaborative and participatory process of measuring capacities in each site<br />Evaluat...
Contributing to evaluation<br />Annual assessments produce a site specific T1T2T3<br />Use additional data to answer que...
Lessons<br />Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of...
Lessons/Challenges<br />Important to anchor discussion of capacities in terms of results/work<br />Capacity for WHAT?<br /...
Measuring…<br />Resident engagement<br />Community engagement<br />Participation<br />Social networks/social capital<br />...
Authentic Demand “Honeycomb”<br />Residents advocate for results in strategy work groups and play key decision making role...
What Does Authentic Demand Look Like?(The Forms)<br />Leadership strategies that offer learning opportunities and training...
What Does Authentic Demand Look Like?(The Forms)<br />Community organizing efforts that mobilize community members in the ...
How do we know that it’s meaningful?(Interim Outcomes)<br />Voice. There are ample opportunities for community members to ...
3. 	Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, men...
5. 	Reciprocity.  Community members are actively encouraged to contribute their time and talent, because there is a clear ...
Tools & ways of measuring<br />Software for tracking program, process, and contextual data in a central place (Backpack- B...
Tools & ways of measuring<br />pre/post...or often better...post-then assessments of learning<br />charting timelines<br /...
Periphery<br />Cluster<br />Link<br />Node<br />Social Network Analysis: A Few Helpful Definitions<br />Core<br />Hub<br /...
Lessons learned<br />
Other resources<br />Race Matters Toolkit<br />http://www.aecf.org/racematters.aspx<br />Guide to Evaluating Advocacy & Po...
Upcoming SlideShare
Loading in …5
×

Ccdo Feb25 Evaluating Cc Is

578 views
456 views

Published on

Presented at national Weed & Seed CCDO Conference in Baltimore. Feb 25, 2010

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
578
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ccdo Feb25 Evaluating Cc Is

  1. 1. Evaluating community Evaluating collaborationEvaluating capacity<br />Tom Kelly Annie E. Casey Foundation <br />Tanja Kubas-Meyer Making Connections Providence<br />www.aecf.org public.me.com/tomkellyjr<br />tkelly@aecf.org slideshare.net/tomkaecf<br />(410) 223-2932<br />
  2. 2. Annie E. Casey Foundation<br />Established in 1948 by Jim Casey, founder of UPS <br />Casey “cousins” with Casey Family Program, Marguerite Casey Foundation, Jim Casey Youth Opportunity Initiative <br />Mission: To improve the lives of America’s most vulnerable children<br />Operates foster care as Casey Family Services<br />
  3. 3. Annie E. Casey Foundation<br />Began grantmaking in 1980’s<br />State child welfare reform<br />Juvenile detention reform<br />New Futures – community/schools<br />Grantmaking in the 1990’s<br />Urban workforce – Jobs Initiative<br />Urban children’s mental health<br />Teen pregnancy prevention – Plain Talk<br />Rebuilding Communities Initiative<br />Grantmaking in the 2000’s<br />Community change initiative – Making Connections<br />PRI’s and responsible redevelopment – East Baltimore<br />
  4. 4. Overview of Comprehensive Community Initiatives (CCIs)<br />Goals focus on multiple levels, including individual/family, neighborhood, organizations, community, region<br />Focus on building capacity of neighborhood stakeholders and local institutions<br />Attend to interrelated influences <br />Multiple players, layers, systems<br />
  5. 5. Challenges in Evaluating Comprehensive Community Initiatives:<br />Outcomes are either too distal (reduced poverty rates) or too abstract (improved community capacity)<br />Difficulty developing a counterfactual (e.g., Community wide intervention, other programs tackling same issue, lack of control group or comparable community)<br />Outcomes and theories of change shift, as community needs fluctuate and learning occurs<br />Unexpected outside influences<br />
  6. 6. Range of Tools for a Continuum of Data Needs/Questions<br />Outcome Evaluation<br />Surveys<br />Anecdote<br />Observations<br />Random Control<br />Comparison<br />T1 vs T2<br />Sampling<br />Focus Groups<br />Gut<br />Guess<br />(no info)<br />Strongest <br />evidence<br />Good Enough<br />to Decide/Act<br />Now<br />
  7. 7. Measuring Success<br />Change in PLACE<br />Change in PEOPLE<br />Change in COMMUNITYCAPACITIES<br />Change in SYSTEMS OF SUPPORT & SERVICES<br />Change in ENVIRONMENT & CONTEXT<br />Contribution of Weed & Seed - Explain WHAT, HOWand WHEREthe initiative is making a difference<br />
  8. 8. RBA Model (in one slide)<br />Who are we concerned about? Population<br />What conditions or change do we want for this population? Result<br />How can we measure these results? Indicators<br />How would we know if we are making progress? Performance measures<br />Who has a role to play? Partners<br />What works? Best practices<br />What do we propose to do? Our theory and strategies<br />
  9. 9.
  10. 10. Results…what difference did we make in terms of:<br />IMPACT on children and families<br />INFLUENCE on policies, practice, systems, and services<br />LEVERAGE of resources<br />
  11. 11. Impact, Influence, Leverage<br />We seek to make a positive difference in the lives of children and families through three types of RESULTS:<br /> Impact: Changes in a condition of well being for the children, adults, families or communities directly served by grants, programs, agencies, or service systems.  Influence: Changes in capacity (including cultural competence), policies, regulations, systems, practice or public opinion. Leverage: Changes in investments by other public or private funders in our strategies to improve outcomes for children and families.<br />
  12. 12. Place-based community accountability<br />Program-level AND population-level (whole neighborhood) accountability<br />Beyond traditional contractual obligations to a funder<br />Strengthening shared and collaborative accountability to the whole population across multiple partners and organizations<br />Increased partnership of all partners, players, and funders working in a place<br />
  13. 13. Accountability requires…<br />Funders and decisionmakers (i.e., those with power and resources) are held accountable for the decisions they make and they must publicly affirm and report their targets and results <br />All stakeholders must have publicly available data with which to hold all accountable<br />Community residents must be prepared to be organized and participate and act in decisionmaking and accountability processes<br />Transparent and open decisionmaking processes and structures need to exist to hold all stakeholders accountable<br />Incentives for positive progress and sanctions for negative change<br />
  14. 14. Results Accountability<br />www.resultsaccountability.com<br />http://raguide.org/<br />www.aecf.org “Casey ResultsNet”<br />
  15. 15. Weed & Seed<br />Collaboration<br />Coordination<br />Community participation<br />Leverage<br />
  16. 16. Weed & Seed<br />Collaboration<br />Coordination<br />Community participation<br />AUTHENTIC DEMAND<br />Leverage<br />CORE CAPACITY<br />ASSESSMENT<br />
  17. 17. Measuring Leverage<br />A Practical Guide to Documenting Influence and Leverage In Making Connections Communities<br />Reisman, Jane; Langley, Kasey; Stachowiak, Sarah; Gienapp, Anne<br />www.organizationalresearch.com<br />www.aecf.org<br />
  18. 18. Making Connections<br />Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based strategy<br />10+ year investment in 10 communities (2000 to 2010)<br /> Hartford Indianapolis<br /> Louisville Milwaukee<br /> Oakland Providence<br /> Denver Des Moines<br /> San Antonio Seattle/White Center<br />
  19. 19. Results/Outcomes<br />Adults have work and earnings<br />Families save, build and protect assets<br />Families access quality services <br />Families are connected to social supports<br />Children are healthy and prepared to succeed in school (grade-level reading)<br />
  20. 20. Local Learning Partnerships (LLPs)<br />Established to support local Making Connections sites (for 10-years and beyond) with data and information to help strategy development, and self-evaluation <br />Collaborative group of data holders, researchers, evaluators, and data users (including nonprofits and residents)<br />Multiple roles:<br />Building/consolidating local data warehouses<br />Performance and results measurement and feedback to work teams<br />Quantitative and qualitative data collection on families and neighborhoods, including evaluation of process and outcomes<br />Promoting local capacity (by residents and systems) to use data strategically<br />
  21. 21. Community capacities<br />Program implementation and management<br />Data, self-evaluation, and accountability<br />Communication, public policy, and will-building<br />Resident leadership and engagement (Authentic Demand)<br />Financial sustainability and infrastructure<br />
  22. 22. Community capacities<br />SCOPE – coordinated and integrated services and service systems<br />SCALE – policy and system changes that reach large numbers of families<br />SUSTAINABILITY – financing and infrastructure<br />Authentic demand/resident engagement<br />Data, learning, and accountability<br />
  23. 23. Measuring Community Capacities<br />Began with stages of change framework (INSITES, Bev Parson)<br />
  24. 24.
  25. 25. Building a scale<br />Developed common (general) indicators across sites<br />Worked with sites to revise and better define concepts and stages<br />Tested in individual sites <br />Added a “score”<br />
  26. 26.
  27. 27. Using the scale<br />Implemented collaborative and participatory process of measuring capacities in each site<br />Evaluators facilitated a review and discussion of specific areas of work (eg, workforce, assets, school readiness)<br />What is the evidence?<br />Site specific score<br />
  28. 28. Contributing to evaluation<br />Annual assessments produce a site specific T1T2T3<br />Use additional data to answer questions like:<br />Did technical assistance help sites advance? <br />What patterns exist across sites?<br />Does higher capacity lead to more/bigger/better results for families?<br />
  29. 29. Lessons<br />Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of concepts and theory of change<br />(decision) More important to focus on change within sites than comparing sites given different environments/context<br />
  30. 30. Lessons/Challenges<br />Important to anchor discussion of capacities in terms of results/work<br />Capacity for WHAT?<br />Iterative process<br />Still difficult to summarize across sites<br />Need additional verification of evidence and perspectives<br />Tool/scale less important than the process<br />
  31. 31. Measuring…<br />Resident engagement<br />Community engagement<br />Participation<br />Social networks/social capital<br />Civic engagement<br />Community organizing<br />
  32. 32. Authentic Demand “Honeycomb”<br />Residents advocate for results in strategy work groups and play key decision making roles<br />Membership networks continue to grow in size and effectiveness<br />Resident leadership training institutionalized in community colleges and other local partners<br />Organizing leads to policy changes at local and state level<br />Resident leaders provide impetus for forming service provider networks<br />
  33. 33. What Does Authentic Demand Look Like?(The Forms)<br />Leadership strategies that offer learning opportunities and training to residents and other partners focused on various skills and capacities including strategic planning, data analysis, policy advocacy and others; <br /> <br />Strong social networks that build relationships and reciprocal exchange among and between members of the community, and link residents to new resources and opportunities;<br />
  34. 34. What Does Authentic Demand Look Like?(The Forms)<br />Community organizing efforts that mobilize community members in the collective and in strategic alliances with other stakeholders seeking better outcomes for children, families and neighborhoods<br /> <br />The kind ofcivic participation that enables residents to establish community priorities and then hold elected officials and service providers accountable through voting, local forums, community research, and consumer and policy advocacy. <br />
  35. 35. How do we know that it’s meaningful?(Interim Outcomes)<br />Voice. There are ample opportunities for community members to not only make their needs, desires and opinions known, but also to participate with dignity in environments where decisions affecting the community are being made. <br />Accountability. Those who represent community interests at decision-making tables are representative of and accountable to well-defined and legitimate constituencies, including voters, members of a community or faith-based organization, or agency leadership. As important, there is evidence of a “demand environment” services, resources and opportunities are determined more by the needs and desires of those in the community rather than what organizations have available or what service providers “decide” families and the neighborhood need. <br />
  36. 36. 3. Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, mentoring, or other skill building opportunities that enhance the competence and confidence of community members at decision-making tables and in civic life. <br />4. Identity. People believe they hold a stake in the larger community. In addition, community identity is inclusive and represents its diversity -- different cultural groups within the community are genuinely represented and embraced as members of the community.<br />
  37. 37. 5. Reciprocity. Community members are actively encouraged to contribute their time and talent, because there is a clear understanding that everyone has something to offer the change agenda. Service to the community is rewarded through public recognition, ceremonies, and events. A social “norm” of participation and of action is nurtured.  <br />6. Choice. Not only are there many options to engage in and contribute to change efforts, but those options connect to the needs and desires of the community – what has value for them – not the agendas of someone else. <br />
  38. 38.
  39. 39.
  40. 40.
  41. 41. Tools & ways of measuring<br />Software for tracking program, process, and contextual data in a central place (Backpack- Backpackit.com)<br />Tracking resident engagement (Constant Contact and YS.net)<br />Mapping participation (GIS)<br />
  42. 42. Tools & ways of measuring<br />pre/post...or often better...post-then assessments of learning<br />charting timelines<br />learning meetings<br />core capacity assessment<br />individual and group interviews<br />social network analysis<br />
  43. 43. Periphery<br />Cluster<br />Link<br />Node<br />Social Network Analysis: A Few Helpful Definitions<br />Core<br />Hub<br />Source: Monitor Institute<br />
  44. 44. Lessons learned<br />
  45. 45. Other resources<br />Race Matters Toolkit<br />http://www.aecf.org/racematters.aspx<br />Guide to Evaluating Advocacy & Policy Change http://www.organizationalresearch.com/publications/a_guide_to_measuring_advocacy_and_policy.pdf<br />When and How to Use External Evaluators<br />http://www.aecf.org/publications/data/using_external_evaluators.pdf<br />Tools and Resources for Assessing Social Impact (TRASI) (beta)<br /> foundationcenter.org/trasi<br />

×