Innovative Synthesis Methods


Published on

Learn key steps to synthesize numerous evaluation studies that have different target groups, interventions, evaluation designs, statistical techniques and geographical locations. Utilizing principles of the realist synthesis method and multiple case studies contributes to innovative alternatives to the meta-analysis technique.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Limitations: Most of the evaluations had only completed three quarters of the implementation to date. (These are not based on final reports)
  • Discuss the possibilities of using meta-analysis
  • All good synthesis methodology should include these types of triangulation.
  • The NCPC had questions related to Knowledge, Attitudes, Skills, and Risk And Protective factors, however, for the purposes of this presentation we are only sharing questions related to behavioural change
  • Evaluation studies are expected to measure pre, post-program, 6 month and 1 year follow up of project participants
  • E =experimental group C=comparison group
  • To determine the % of sub-measures that were favorable, the following calculation was used: (# of favorable sub-measures/Total # of sub-measures in each table) To determine the % of projects that demonstrated favorable change in each table, the following analysis was conducted. Calculate the most frequently occurring result (i.e favorable, unfavorable or no change result) for each unique program. The result that occurred most frequently determined the result for that program. For example, if 1 program had 4 sub-measures related to anger, with 3 of the 4 sub-measures showing a favorable change, that program would be assigned the “favorable label”. Programs that had an equal ratio of favorable to unfavorable or no change result were excluded from the sub-analysis as an accurate method to identify one label could not be identified. An overall calculation was conducted as follows: # of favorable program results/Total # of program results The % of projects demonstrating a favorable change for the short term, intermediate and long term could be achieved using this method. The appendix section provides reference information for details about the individual studies. Further details explaining how the sites are selected for an impact evaluation are also included in the Appendix section.
  • In the NCPC case study, the only common variables identified included the following concepts: Favorable change: a result that demonstrates a statistically significant (p<0.05) or clinically significant positive change Non-Favorable change: a result that demonstrates a negative change in the outcomes being measured No Change: a result that does not demonstrate a statistically significant change between the pre and post test follow up periods
  • Note the sample size for youth having behavioural changes is smaller. This would explain the relatively higher percentage of projects demonstrating reductions. There are approximately 33 outcomes (indicators) that form the basis for the immediate outcomes and approximately 6 outcomes that form the basis for long term outcomes. The relatively smaller number of long term outcomes demonstrated more favorable change and that is why there is a relatively more favorable change when compared to the intermediate outcomes Intermediate outcome examples include: substance abuse, employment, family attachment, withdrawal, aggression, anger management, rule breaking, self-esteem Long term outcomes involve: police contact, arrests, victimization non-violent offending and violent offending
  • Fortunately we have some structure for categorizing effect sizes and eta-squared: Cohen d 0.2=low effect size 0.5=moderate effect size 0.8=large effect size Eta-Squared (ANOVA-if F ratio is statisticialy ) .09=low strength and magnitude .14=moderate strength and magnitude .22=large strength and magnitude Eta-squared value reflects the strength and magnitude related to a main or interaction effect. Eta-squared is very similar to r squared in a regression equation (it indicates the amount of contribution the variables or intervention has in the outcome)
  • Innovative Synthesis Methods

    1. 1. Innovative Synthesis MethodologiesDonna Smith-MoncrieffeCanadian Evaluation Society ConferenceFairmont Royal York, TorontoJune 9-12, 2013
    2. 2. 2Summary of the Presentation NCPC Mandate Government Context Evaluation considerations at the corporate level (Performance) Evaluation considerations at the project’s level (Knowledge products) Overview of 3 Types of synthesis methods Key points Challenges Case Study (Step by Step Process) Collating data from projects with different target groups, interventions and outcomes Synthesis tables Providing reporting options for summarizing various statistical results: percentages,Effect Sizes, T tests and F coefficients Conclusions
    3. 3. 3
    4. 4. Government Context: Evaluation Approaches4
    5. 5. Why did we need to Synthesize the Data?● We needed to respond to the questions about, “what works in crimeprevention”● We needed to provide findings and conclusive statements about 13programs (total of 36 project sites) with different:- Target groups;- Interventions;- Outcomes; and- Evaluation designs● Single site repeated design● Quasi-experiment design (sometimes with a matchedcomparison group)5
    6. 6. Types of Synthesis Methods6
    7. 7. Synthesis Method Requirements for qualityand Rigor7
    8. 8. What approach did we use to Synthesize 13Evaluations?8
    9. 9. Step 1: Identify Key QuestionsWhat behaviour related changes were made (i.e. changes in violentoffending, police contact and victimization?­ What percentage of projects made favorable changes in relation tothe criminal justice system (i.e. reduced violent offending, policecontacts etc..)?­ How much change was made?­ What contextual and program related factors contributed to thesechanges (i.e. geographical location, partnerships, resources etc..)?9
    10. 10. Step 2: What are the inclusion criteria?● In the Government context, we need to use as many evaluation studiesas possible in the synthesis; even when they do not meet rigidevaluation standards.● Evaluation studies that met the following minimum requirements wereincluded in the synthesis:­ Study uses quantitative data (descriptive or inferential statistics);­ Measures that correspond with key crime related outcomes (i.e.association with anti­social peers, emotional regulation, substanceabuse, offending, education, employment etc..);­ Study used at least the basic evaluation design (single group pre­post test repeated measures design); and­ At least pre and post test measures had been available10
    11. 11. Step 3: Enter the Data (Partial Sample)Name ofStudyDescriptionof VariableSampleSizeResult P Level Type ofChangeLevel ofStudyProjectInterventionTorontoArrests E=76C=43F=0.291 P=.823 NoChangeExemplaryMulti-SystemicTherapyYouth notarrested foran offenceE=28C=12E=89.3%C=66.7%N/A Favorable BasicTowards NoDrugsWeaponCarryingE=847C=54F=0.63 P<0.01 Favorable NotableVelocity PolicecontactsE=87 69%(reduction)N/A Favorable NotableInterpretation of Results11
    12. 12. Step 4: Analysis and InterpretationSynthesis for 9 evaluation studies measuring changes in criminalbehavioursOverall interpretationBackground Information•Number of evaluation studies: 9•Number of sites: 11•Common measures include: 1) Arrests; 2) Non-Violent Offending; 3) Violentoffending; 4) Criminal Victimization; and 5)Weapon CarryingQuantitative results•% of change by sub-measure: 50%•% of projects demonstrating favorable change (showing at least one stat.significant result at the project level: 75%•% of projects demonstrating unfavorable change:0%•% of projects demonstrating no change: 25%Qualitative results•Provide qualitative results by model, site or project level to qualify the quantitativefindings12
    13. 13. Step 5 : Reporting Aggregate Data13
    14. 14. Step 5: Reporting Aggregate Data(Sample of Approach # 1)14
    15. 15. Step 5: Reporting (Sample of Approach #2reporting on magnitude of change )15
    16. 16. Conclusions● What synthesis method works best when the evaluation studies have avariety of interventions, outcomes and statistical measures?­ Elements of the Realist Synthesis review and Multiple­case StudySynthesis are recommended; and­ NCPC utilized elements of both of these synthesis methods but couldbenefit from responding to the Why questions incorporated in theMultiple­ Case Study Synthesis● What reporting method works best to summarize non-standard data?­ Approach #1 and # 2 are both useful; and­ Assess your audience to determine the best reporting method16
    17. 17. Conclusions cont’d● What planning activities would be required to utilize both Realist andMultiple Case study Methods?­ Develop strategic direction (strategic plan to help focus the direction ofthe evaluation);­ Prioritize key areas to minimize complexities;­ Ensure that the information management system is designed to:● Clarify the unit of analysis (by model, by project, evaluationquestions, etc..);● Respond to How and Why questions such as: How does theprogram produce these effects or How does the interventioncontribute to change?; and● Collect data about the context for each model, site or project.17
    18. 18. References for Recommended SynthesisMethods18
    19. 19. 19Thank YouContact:Donna Smith­Moncrieffe, BSc. Crim Dip, MSc.Senior Evaluation AdvisorPublic Safety Canada, National Crime Prevention CentrePolicy, Research and EvaluationTel: 416­952­0423Fax: 416­952­0483Email: donna.smith­moncrieffe@ps­sp.gc.caWeb : |