Recommended
Recommended
More Related Content
What's hot
What's hot (11)
Similar to Ready by 21: Using Data in Coalitions
Similar to Ready by 21: Using Data in Coalitions (20)
Recently uploaded
Recently uploaded (20)
Ready by 21: Using Data in Coalitions
- 1. Using data in coalitions: Moving from reporting to learning Michelle Shumate shumate@northwestern.edu Kate Cooper katherine.cooper@northwestern.edu Twitter: @NNSI_NU nnsi.northwestern.edu
- 2. About Us… 2
- 3. Does this sound familiar? 3 “We’ve collected a lot of data [in 2.5 years]. We haven’t issued a baseline report…we’re at a point where we could issue [a report], but we haven’t done that.” - North Carolina
- 4. How about this? 4 “[Data collection] is a rabbit hole. We’re building the bicycle while riding it.” - Illinois
- 5. Or this? 5 “You can look at the data but you don’t know what it means…there was some concern about data being released to the community without someone to explain what it means.” - Maine
- 6. But when it’s working … 6 ““[We created] a HIPPA compliant data system that allowed us to serve families in real time, no matter which agency they're working for. On a monthly basis, actually spits out reports on each case manager’s performance. If the case managers are meeting the performance benchmarks, then the system is working.” - California “The communities receive data from three sources: from students, the partners, and from the National Student Clearinghouse. [Our network] sees that data and aggregates it. I don’t get to see the name of each individual…just the aggregate and the patterns so we can see what’s working and what’s not across communities.” - Pennsylvania
- 7. Maybe we can all agree on this… “It’s a really, really long process to get to data sharing…” Washington State 7
- 8. Goals and Introductions Goals • To explore how your community uses data to improve outcomes for youth • Identify strategies for your community to collect and use data in decision-making 8
- 10. 10 How does YOUR community use data? Compliance ProfessionalNegotiation
- 11. Measuring Success • In traditional program evaluation (when an organization evaluates its own programs), how do you know if you’ve succeeded? • In collective impact, how might this differ? 11
- 12. The Journey to Continuous Network Improvement Establish common outcome and program metrics. Develop a shared system for collecting and comparing information. Convene learning community or workgroups. Set SMART goals, track progress, be accountable, and continue to refine programs. 12
- 13. Getting Started: What do you need to share? Do you have: • Shared definitions for measures? • Data-sharing agreements? • Plan for data sharing – process? – Plans for cleaning/formatting – Tool for sharing/storing (e.g., repository, software) – Plan for distributing (frequency, discussing) 13
- 14. 14 Goodhardt’s Law: "When a measure becomes a target, it ceases to be a good measure.” Leading Indicators Lagging Indicators % students attending a high quality preschool Kindergarten readiness % of students with excessive absences 3rd grade reading % of students that participate in co- curricular, extra-curricular, or out of school program related to vocational career path Complete meaningful post-secondary education program
- 15. Activity goal Bridging Goal Outcome goal 15 Have 10 outreach events to promote healthy living Target population has a more positive attitude towards healthy behaviors Reduction in obesity All ESL teachers complete 20 hours of continuing learning on new ESL techniques ESL teachers use more effective techniques in the classroom based on peer evaluation. Improved literacy outcomes for ESL students.
- 16. Getting Started: What do you need to share? Plan for data sharing • Who has access? • Who analyzes? • How are decisions made? • How to evaluate your efforts? 16
- 17. 17
- 18. 18
- 19. What is different about CBC? • Use of funding to incentivize individual organizations • No joint programs or policy efforts – Emphasis on individual goals 19
- 20. 20 Follow-Ups Recorded in Spring 2015 Parents Who Read With Their Child Daily for 20 Minutes Parents Who Have Increased the Frequency of Reading They Do With Their Child Parents Who Increased Their Frequency of Reading or Read to Their Child Daily for 20 Minutes # of Survey Respondents # % # % # % Children’s Home + Aid Society of Illinois 32 30.48 32 30.48 105 Chinese American Service League 1 3.33 13 43.33 13 43.33 30 Christopher House 27 23.00 47 40.51 58 50.00 116 Erie Neighborhood House 15 13.00 32 27.82 42 36.52 115 Gads Hill Center 16 17.00 33 35.49 38 40.86 93 Totals 59 12.53 162 34.39 188 39.92 471 Exhibit 3: Outcome Results for Parent Engagement (Reading to Their Children)
- 21. 21 Fall 2014 Spring 2015 Average Change Showed 0.5 Increasea Showed Any Increase # of Program Participants Children’s Home + Aid Society of Illinois 0.92 1.50 0.58 39.1% 78.3% 46 Chinese American Service League 1.00 1.90 0.90 100.0% 100.0% 5 Christopher House 0.85 1.52 0.67 58.8% 94.8% 97 Erie Neighborhood House 1.58 2.23 0.65 55.0% 80.0% 20 Gads Hill Center 1.35 2.67 1.31 100.0% 100.0% 9 Overall 1.14 1.96 0.82 70.6% 90.6% 177 Exhibit 4: Outcome Results for School Readiness: Literacy Indicator Outcome: Increased school readiness (ages 0–3)
- 22. 22 Fall 2014 Spring 2015 Average Change Showed 0.5 Increase Showed Any Increase # of Program Participants Children’s Home + Aid Society of Illinois 2.98 4.40 1.42 87.8% 97.8% 90 Chinese American Service League 2.63 3.32 0.69 59.3% 92.6% 27 Christopher House 3.24 4.46 1.22 84.3% 95.9% 197 Erie Neighborhood House 3.20 4.07 0.87 77.2% 96.7% 123 Gads Hill Center 2.96 3.88 0.92 73.3% 90.7% 75 Overall 3.01 4.03 1.02 76.4% 94.7% 512 Outcome: Increased school readiness (ages 3–5)
- 23. Small group activity 23 Read the short vignette provided and answer the questions in the last paragraph.
- 24. 24
Editor's Notes
- Background in research, nonprofits/collaboration Legitimacy building - capacity project, progression to research in coalitions
- What is your definition of what you’re measuring? - How do you work out discrepancies between partners? Data-sharing agreements Legally, an organization has what they need to share Likely means “scrubbed” data and use of identifiers rather than names Likely means drafting & signing MOU Plan for actual data-sharing Cleaning and formatting – what form does the data need to be in? Repository? Software? Add in a little bit about software – case management – Sales Force, Efforts to Outcomes Distributing – how often are you supposed to upload and review
- Who has access? Who provides the data, who’s responsible for knowing it, who has access to any particular system Who analyzes? Address the need for someone who has some statistical knowledge Ability to look across organizations/systems to make comparisons Can develop more complex analytics Example of materials – StriveTogether uses Lean Six Sigma; what do others do? How are decisions made? How often? By whom? Who’s responsible for understanding the data from each organization? How to evaluate your efforts? collective impact initiatives should evaluate data and the backbone Assuming you have these things in place – a plan and process for sharing – can start working through it as described in this exercise
- Information on why CI initiatives collect data Reminder of how this structure is different from others - to align and evaluate what is and isn’t working versus creation of new programs Data that suggests impact rather than outcomes Can bring in some of Benn’s 6 purposes of evaluation Our experience: talk about the distinction we see between what people say they’re doing and what they actually do Visual option: have some key phrases/quotes from EC2C and Lodestar interviewees to talk about data to represent on slides What we see in benchmark reports
- In Exhibit 3, the first column represents the percentage of parents who achieved the goal of reading to their child for 20 minutes a day. Which agency had the greatest percentage? Christopher House was highest, reporting that 23% of its parents read to their child at least 20 minutes a day. In Exhibit 3, the second column represents the percentage of parents who have been increasing their reading but are not yet at the goal of 20 minutes daily. Which agency showed the greatest gains there? The Chinese American Service League showed the greatest gains, with 43.33% of parents increasing their reading time. Christopher House was close behind at 41%. One of the interesting things to note is the Chinese American Service League is heading toward the goal that it had very little success with in the past, as shown by the fact that only one of its parents had read to their child every day for 20 minutes. In Exhibit 3, the third column is the combination of those who reached the goal and those moving toward it. Which agency is struggling the most in both? The Children’s Home + Aid Society is struggling most, with only 30.48% of its 105 surveyed families either increasing their reading to their child or achieving the goal of 20 minutes of reading a day. The agency is in a good position to learn from other CBC agencies who are getting better results and to set a SMART goal for improvement in the next academic year.
- Taking a look at Exhibit 4, what kind of metric is the literacy indicator for school readiness? Is it a leading or lagging indicator? School readiness is a lagging indicator. The key idea here is that you can directly affect parental reading to children through a program, but you can’t directly affect literacy. It’s the result of an increase in another activity. Exhibit 4 compares students’ performance on literacy assessments over the course of the school year, from fall 2014 to spring 2015. There are three measures of change. The first two columns display the average scores from fall 2014 and spring 2015. The third column shows the average change over that time period. The fourth column indicates the percentage of children that showed at least a 0.5 increase, and the fifth column indicates the percentage of children that showed any increase. Whereas small increases might be due to maturation, the working theory is that a 0.5 increase is significant. Which agency had the greatest percentage of students with at least a 0.5 increase for ages 0–3? How do you make sense of the 100% scores for the Chinese American Service League and Gads Hill Center? For ages 0–3, 100% of Gads Hill Center and Chinese American Service League students showed a 0.5 increase or more in their literacy levels. One of the things for students to ponder is how much weight to give the 100% scores for the Chinese American Service League and Gads Hill Center for ages 0–3. They should note the small number of students in those groups: only five in the Chinese American Service League and nine in the Gads Hill program. With these small numbers, these programs may not be emblematic of best practices, but just provide lots of individualized attention. It’s noteworthy that these same two programs have the lowest percentage of students making gains of at least 0.5 for ages 3–5, where larger numbers of students are involved
- What about ages 3–5? For ages 3–5, Children’s Home + Aid Society had the largest percentage of students with at least a 0.5 gain (87.8%).
- Christopher House Breakout Group(s) report What’s the right narrative to accompany these data? Christopher House directors may be tempted to focus on just the agency’s strengths rather than its weaknesses. Although there are bright spots, like the percentage of children ages 3–5 who are making significant gains in reading, there are also concerns. For example, only 23% of parents are reading to their child for 20 minutes every night. Further, only 58.8% of children are making significant gains in the years 0–3. This suggests that there are both encouraging signs and opportunities for growth. What might make Christopher House’s data more or less comparable to those of the other CBC partners? Students will typically report that these data might be measured in different ways and so they are not strictly comparable. However, the case explicitly mentions that the CBC partners used the exact same measures, so it is important to correct them on this point. Other factors might include the degree to which different agencies only serve low-income students or whether Christopher House serves a greater number of students who speak English as a second language. Are changes in the literacy indicator for school readiness linked to changes in parents reading to their children? The charts provided in the case do not prove such a link, but the data-sharing system that CBC uses allows the member agencies to statistically analyze whether children whose parents made gains in reading to them every night also increased their literacy the subsequent year. Having a data system that tracks both leading and lagging indicators at the client level makes that possible. Chinese American Service League Breakout Group(s) report How should you interpret the results? Can you rely on the data related to parent reading? Do the data accurately reflect parents’ reading engagement? One key consideration in determining the comparative accuracy of parental reading rates is response rate to the parent survey. In particular, Christopher House’s survey response rate is relatively lower than the others, at 40%. Students could determine this by comparing Christopher House’s total number of children in Exhibit 4 (294) with its number of survey respondents in Exhibit 3 (116). Lower rates of response lend themselves to questions of response bias: Were parents who made no gains less likely to complete the survey, leading Christopher House to inflate its success? A second concern is the accuracy of the parents’ self-reported behavior. There may be a tendency for some parents to report what they had hoped to do rather than what they actually did. How can the data be pulled together to develop SMART goals? SMART goals are only as good as the data you collect. With faulty data, both the evaluation of the baseline and realistic progress are undermined. Erie Neighborhood House Breakout Group(s) report How do you make sense of the discrepancy between the leading (parent engagement) and lagging (literacy) indicators? There are a couple of ways students might go with this answer. Some student groups will focus on the difficulty of ensuring the accuracy of the leading indicator data. Others will question the logic of the theory that parents reading to their children is a determining factor of school readiness. A number of other factors could affect a child’s school readiness, including instructional quality in the classroom and unique challenges their population of students faces. Parent engagement alone may not be enough to overcome these issues. How can a SMART goal and plan for improvement be developed, taking the data into account? Students should be encouraged to think about a couple of factors. The first is a time lag, or how immediate is the impact of the leading indicator change on the lagging indicator. If this is the issue, then the data should show the correlation with parent reading the following year. The second is whether parent engagement is the right lever for Erie Neighborhood House. Because its students start with relatively high literacy scores in comparison to other agencies, maybe another intervention is needed. In this case, the SMART goal wouldn’t focus on parent reading at all.