Show me the evidence
•
2 likes•313 views
Jessica Weitzel presented “Finding and Incorporating Research to Increase Program Effectiveness” the training was sponsored by the After-School Network of Western New York [@asnwny] and held at the United Way of Buffalo and Erie County [@uwbec].
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Show me the evidence
Similar to Show me the evidence (20)
Recently uploaded
Recently uploaded (20)
Show me the evidence
- 1. So you can see the money to help kids and families Aka: Finding and Incorporating Research to Increase Program Effectiveness
- 3. Why do we need to do this? Where can we find good research? How do we incorporate it into our program? How do we know if what we do is working?
- 4. Why do we need to do this?
- 5. Prove that we are making a difference!
- 6. Funding
- 8. Applications must demonstrate that the eligible entity will use best practices, including research or evidence-based practices, to provide educational and related activities that will complement and enhance academic performance, achievement, postsecondary and workforce preparation, and positive youth development of the students.
- 9. Applicants will be awarded up to 12 points for providing evidence that their proposed intervention will lead to the outcomes identified in the logic model. Applicants shall provide a description of up to two research studies or valuations that provide evidence that the proposed intervention is effective for the target population and community problem, and should describe how this evidence places them in the highest evidence tier for which they are eligible.... More points are awarded for higher tiers of evidence.
- 10. Using, generating, and sharing evidence about effective strategies to support students gives stakeholders an important tool to accelerate student learning. ESSA emphasizes the use of evidence-based activities, strategies, and interventions (collectively referred to as “interventions”). https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf
- 16. Where can we find good research?
- 17. Who do we reach? What results do we expect?
- 20. How do we incorporate evidence into our program?
- 23. How do we know if what we do is working?
- 28. Inputs Activities Outputs Outcomes Impact Program Monitoring
- 29. The systematic and continual documentation of key aspects of a program in order to assess whether the program is being implemented as intended. Program Monitoring Quality Improvement Process Monitoring Fidelity of Implementation Performance Management Process Evaluation
- 30. Inputs Activities Outputs Outcomes Impact Program Evaluation
- 31. The systematic method for collecting, analyzing, and using information to answer questions about a program’s effectiveness. Program Evaluation Program Outcomes Impact Evaluation Outcome Evaluation
Editor's Notes
- Introductions around the room. Why came today?
- We need to prove we’re making a difference for at least 2 reasons: Funding Participant impact
- Leads to more funding. Programs that can prove that they monitor performance and their impact through rigorous evaluation and use that data to continuous improve have far better chance of getting funded.
- It’s not just in grants... ESSA requires schools/districts to use evidence to make program decisions We have also seen local funders increasingly asking clients, “where’s the evidence?” (Oishei, Cullen, Tower, BPS...)
- But funding isn’t everything – majority of us are in it for the people we serve. And we need to know for certain that we are having the impact that we want to have. It’s not enough to think we are making a difference or try our hardest to make a difference. All programs THINK and WANT to make a difference, and they are doing GOOD things, so thought is “What could go wrong??”
- DARE example – Approximately $1billion, found to be ineffective at any drug resistance outcomes, and counterproductive among some populations with increased drug experimentation and use. There are A LOT of similar examples. BUT all is not lost...
- Keepin’ It Real now used. Lots of examples like this – illustrating the point that you can’t assume that a good-sounding program will have the results that you want. Program evaluation can help ensure that the program is achieving its goals, and if it isn’t, they can go back to the drawing board. DARE now uses Keepin’ It Real and has reformulated message and outcomes.
- Not every intervention is as intense as the two just presented. And, fortunately, there is a growing number of databases and resources to help you find evidence-based programs and research related to the age groups you work with and outcomes you’re trying to achieve. Also pay attention to WHAT DOESN’T WORK
- First, you need to know clearly what your program is trying to achieve and who you are planning to reach.
- A browser full of options
- A lot of programs working in recent years to enhance SE Development EBPs have impact on more than one aspect Look at search terms in each case; may not be explicitly “SEL”
- You probably already have a framework. Use evidence-based programs within your framework to make sure the structure is strong.
- Logic model-point out recipients, setting, intended outcomes; take these into account in your building. You will probably build the structure using different interventions for different aspects of the overall program (at least for ASPs)
- EBPs often come with guidance on HOW to implement as well as how to evaluate e.g., Second Step has a pre-post knowledge test
- You probably have more data than you think
- Are we measuring the right things? Are we measuring them well? What is the data telling us? What measures should we use? Fidelity matters? Measures are often a required or suggested part of purchased EBPs
- Example of using a validated measure of your program’s outcomes
- The IF side is generally used to inform program monitoring (or QI, PM, etc). The LM shows what you expect to do, your outputs show what you expect to have happen as a result. This is how you can monitor whether you are implementing the program as intended – are you reaching the right people? Are you offering enough hours? Are enough people attending? ONGOING.
- Basically, measuring and monitoring whether the program is doing what it is intended to do. Not about is it working! Just about whether it’s happening. Ongoing, usually done internally by organizations.
- The THEN side is used to form the foundation for the Program Evaluation, or how you will measure program outcomes. This shows your outcomes, and you will use DATA to answer whether you’ve achieved your outcomes. Usually, annually or at end of program. This is not as continuous as PM. Example: Second Step pre-post knowledge provides short-term outcome information BUT did the program impact behaviors (next step)? Many clients using the DESSA to measure this.
- Assessing the results or impact of a program. This is the “are you making a difference” part. Did you achieve your goals?
- Only in case internet doesn’t work Search in blueprints, WWC, search Google
- Only in case internet doesn’t work Search in blueprints, WWC, search Google
- Only in case internet doesn’t work Search in blueprints, WWC, search Google
- Only in case internet doesn’t work Search in blueprints, WWC, search Google
- In contrast, Raising Healthy Children (formerly Seattle Social Development Project), tons of research behind it. Positive impact on academics, drug use, risky sexual behaviors, etc. LIFELONG positive impacts.