Using a Fidelity Index to Increase Program Attribution in Impact Evaluation Studies     Presented by Donna Smith-Moncrieff...
Presentation OutlineNational Crime Prevention Centre (NCPC) mandateWhat is a Quantifiable Fidelity Index? Why is it Import...
NCPC Mandate/Core ActivitiesMission statement:To provide national leadership on effective and cost-efficientways to both p...
NCPC Priorities:Provide funding to the following target groups/crime issues:  Children and youth at risk  Crime prevention...
Use of Evaluation in NCPC         Disseminate results              and encourage           the province and               ...
What is Program Fidelity ?Fidelity in evaluation is used to describe the extent to which theinitiative/intervention corres...
Benefits of a Fidelity IndexA Fidelity Index:   Lists the elements that contribute to the success of the   program   Makes...
What is a Quantifiable Fidelity Index?A quantifiable fidelity index is a set of measureable items thatdetermine whether th...
How Can a Measureable Fidelity Index Contribute to          Better Program Attribution?  You can confidently answer the qu...
Use of Fidelity in Four NCPC Model Programs     (Based on literature prior to NCPC Implementation)75% of model programs us...
Key Steps in Using a Quantifiable Fidelity Tool to                Increase AttributionSTEP 1: Identify all the key program...
STEP 1:     Identify all key elements using a comprehensive frameworkBased on Dane & Schneider Framework (1998)Develop the...
STEP 2:                 Apply Appropriate MeasuresDurlak and Dupre 2008 reviewed 59 studies (1998-2005) that had sometype ...
STEP 2:    Sample Treatment Intensity Measures          (Stop Now and Plan® Program                            )Sample : Q...
STEP 3: Construct the causal model      (Multiple Regression Analysis sample)A multiple regression equation can be used to...
STEP 4 :           Calculate and interpret coefficientsInterpretationsOverall, we are looking at the marginal contribution...
STEP 4:    Calculation and Interpretation of Coefficients                        cont’dB e c le a r a b o u t h o w yo u g...
Step 4:     Calculate and Interpret Coefficients                 (sample   data only)Y=               Aggressive Behaviour...
Step 4:           Calculate and Interpret CoefficientsY= 5.274 - 0.4487 (CBS) - 0.3334 (CPG) - 0.2448 (FPS)R Square (adj.)...
Step 4 Calculation cont’d                Use Multi-Level AnalysisInclude fidelity scores in the ANOVA or regression equati...
Step 5: Reporting Results              What Really Contributed to the Results?                       High Levels of Fideli...
STEP 5: Determine if Program Outcomes are related to Fidelity                (Towards No Drugs sample)Outcome             ...
SummaryAttempt to utilize 5 categories: Use Implementation, Dosage,Quality, Participant Responsiveness and Monitoring cont...
SummaryConstruct a multivariate analysis model that will incorporateelements of the fidelity tool   Consider using multile...
Contact InformationDonna Smith-Moncrieffe, BSc., Crim Dip, MSc.          Senior Evaluation Advisor            Public Safet...
Upcoming SlideShare
Loading in …5
×

Using a Fidelity Index to Increase Program Attribution

911 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
911
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using a Fidelity Index to Increase Program Attribution

  1. 1. Using a Fidelity Index to Increase Program Attribution in Impact Evaluation Studies Presented by Donna Smith-Moncrieffe Canadian Evaluation Society-Ontario Chapter October 4th-5th, 2010
  2. 2. Presentation OutlineNational Crime Prevention Centre (NCPC) mandateWhat is a Quantifiable Fidelity Index? Why is it Important?How can using a Quantifiable index increase ProgramAttribution?Steps: From Fidelity Tool Development to interpretationsregarding program attributionSample Fidelity tool and output data for two Model programs Towards No Drug Program (results) Stop Now and Plan Program ® (application of treatment intensity measures)Summary 2
  3. 3. NCPC Mandate/Core ActivitiesMission statement:To provide national leadership on effective and cost-efficientways to both prevent and reduce crime by addressing knownrisk factors in high risk populations and placesCore activities: Supporting targeted interventions in local communities Building and sharing practical knowledge with policy makers and practitioners 3
  4. 4. NCPC Priorities:Provide funding to the following target groups/crime issues: Children and youth at risk Crime prevention in Aboriginal communities Prevent recidivism among high-risk groups Priority crime issues (youth gang, drug-related crimes) 4
  5. 5. Use of Evaluation in NCPC Disseminate results and encourage the province and NCPC Time-Limited Funding municipalities to replicate effective programsResults contribute to Identify and encourage the development of Treasury Board Model and Promising Programs Reports Encourage the Use development of Multi-site Fidelity Tools to Evaluations Increase To Increase Attribution Knowledge 5
  6. 6. What is Program Fidelity ?Fidelity in evaluation is used to describe the extent to which theinitiative/intervention corresponds to the originally intended programThe literature uses the following terms interchangeably Adherence Compliance Integrity 6
  7. 7. Benefits of a Fidelity IndexA Fidelity Index: Lists the elements that contribute to the success of the program Makes the important elements of the program visible and helps evaluators “bound” the program When expected results are unfavourable, model programs can attribute the results to low fidelity levels while still being able to maintain a reputation for being an effective program Identifies what elements practitioners should focus on to increase compliance Ensures that elements in the index will have a complementary quality assurance protocol and definitions for practitioners to review 7
  8. 8. What is a Quantifiable Fidelity Index?A quantifiable fidelity index is a set of measureable items thatdetermine whether the program elements have been implementedas planned. Each item provides a quantifiable measure using ratio or interval level data that can be later used to test for statistical significance, changes in effect sizes or clinical significance The items in the index can be checked for construct validity. We can confidently determine if we are measuring what we think we are measuring. (i.e. Using Chronbach’s alpha and other reliability tests can be achieved by using quantitative measures) Composite indices and aggregate scores can be inserted into multivariate analysis to determine whether the fidelity related scores contribute to participant related changes. 8
  9. 9. How Can a Measureable Fidelity Index Contribute to Better Program Attribution? You can confidently answer the question, ‘Did the model program contribute to change or were additional unintended program elements contributing to changes ? (i.e. Was the inconsistent implementation of cognitive behavioural sessions responsible for the lower than expected changes in anger for boys in the SNAP program?) Each element that is quantified can be placed into a multivariate causal model. The evaluator can: Isolate whether the program’s adherence levels contributed to favourable or unfavourable change in the outcomes of interest Isolate what specific aspect of the program contributed to greater effects or changes in the outcomes of interest Identify the need to find other explanatory variables that are contributing to program impact (i.e. R squared adjusted result < .80 informs us that there are other factors that account for the change in reduction of drugs or offending for example) Identify if low fidelity to the program is contributing to variation in the variables of interest (i.e. Does low participant responsiveness in 25% of the TND classrooms affect the overall goal of preventing or reducing drug use in youth?) 9
  10. 10. Use of Fidelity in Four NCPC Model Programs (Based on literature prior to NCPC Implementation)75% of model programs used 25% of model programs used fidelity fidelity for for Process evaluation only Outcome/Impact Evaluations Method of Attribution is strongerMethod of Attribution is limited to and can identify how much Inferences about how Correlation exists between the Program is related to outcomes Program elements and outcomes 10
  11. 11. Key Steps in Using a Quantifiable Fidelity Tool to Increase AttributionSTEP 1: Identify all the key program elements using a comprehensive frameworkSTEP 2: Apply appropriate measuresSTEP 3: Construct the causal modelSTEP 4: Calculate and interpret the coefficients (multivariate analysis)STEP 5: Reporting: Enhance program attribution. Determine if program fidelity levels contributed to outcomes 11
  12. 12. STEP 1: Identify all key elements using a comprehensive frameworkBased on Dane & Schneider Framework (1998)Develop the fidelity tool and include all elements in the following four areas:1. Implementation: Was the program delivered as intended? (i.e. Were all 8 sessions delivered on a weekly basis over a 2 month period?)2. Dosage: How much of the program was delivered (i.e. Did the youth receive the 48 hours of individual case management?)3. Quality: Are the main components of the program delivered clearly and correctly?4. Participant Responsiveness: Does the program stimulate interest among the participants and practitioners? (i.e. Are teachers interested in implementing the curriculum to students?)5. Monitoring Control/Comparison fidelity- What services did the comparison/control group receive? Did the program adhere to their intention to provide minimal services or “usual care”? 12
  13. 13. STEP 2: Apply Appropriate MeasuresDurlak and Dupre 2008 reviewed 59 studies (1998-2005) that had sometype of fidelity tool and found the following typical measures: 27/59 studies (46%) used categorical measures Ordinal scales (i.e. assigning a definition to low, moderate and high fidelity levels) 32/59 studies (54%) used continuous variables/interval measures Averages reported as percentages (i.e. observations of each session identified that 65% of the youth showed interest in the counselling sessions) Likert scales results were converted to continuous variables Used actual # of hours, # of sessions, # of months especially for dosage related measures 13
  14. 14. STEP 2: Sample Treatment Intensity Measures (Stop Now and Plan® Program )Sample : Quantifiable fidelity checklist (note: this is not the full checklist)vTarget Group Population Met: 8/10v 12 Cognitive Behavioural group Sessions 10/12 Sessions delivered weekly within 3 months Delivered by a trained facilitator Approved manuals used (weekly documentation)v 12 Cognitive Behavioural concurrent parenting sessions 10/ 12Fidelity Level: High T Score: 28/ or 82.4% otal 34____________________________________________________________Fidelity LegendNo Fidelity: 0-29%Low Fidelity: 30-69%Moderate Fidelity: 70-79%High Fidelity: 80-100% 14
  15. 15. STEP 3: Construct the causal model (Multiple Regression Analysis sample)A multiple regression equation can be used to make estimatesabout key program outcomes (i.e. recidivism) based on givenvalues for a number of explanatory variables (i.e. levels of druguse, # of hours participating in treatment, and type ofpractitioner)Ensure all statistical assumptions are met before using the modelConstruct the causal model by using the following equation : y = β0 + β1X1 + β2X2 + …. + β kXk + ε 15
  16. 16. STEP 4 : Calculate and interpret coefficientsInterpretationsOverall, we are looking at the marginal contribution that each X(explanatory) variable is making on the Y (outcome) variable when the other X variables are being held constant.Review and interpret the output and consider the following questions: Are the elements from the fidelity tool statistically significant? What elements of the fidelity tool have higher values and contribute more to change in the key outcomes of interest? What is the overall contribution of the program elements on the outcome? How much does the impact of fidelity have on program outcomes? 16
  17. 17. STEP 4: Calculation and Interpretation of Coefficients cont’dB e c le a r a b o u t h o w yo u g o t fro myo ur fid e l ity to o l to d isc us si ngw h e th e r a d h e re n ce le v e ls h a d a nim p a ct o n p r o gr a m o u tc o m e s …Th e re a r e n o m ir a cle s h e r e … .. “I th i n k y o u s h o u ldb e m o re e xp lic it h e re in s te p tw o .” 11 17
  18. 18. Step 4: Calculate and Interpret Coefficients (sample data only)Y= Aggressive BehaviourConstant 5.274Coefficient of (CBS) Cognitive Behavioural Sessions (0-12 hours)X1 0.4487Coefficient of (DSH) Duration of Service Hours (total program hours)X2 0.3334Coefficient of (FPS) Levels of Fidelity (Participant Responsiveness)X3 0.2448R 2 (adj.) 0.7553 18
  19. 19. Step 4: Calculate and Interpret CoefficientsY= 5.274 - 0.4487 (CBS) - 0.3334 (CPG) - 0.2448 (FPS)R Square (adj.) =.7553 For every increment of cognitive behavioural sessions, aggressive behaviour decreases by 0.4487 For every increment of cognitive parental group sessions, aggressive behaviour decreases by 0.3334 For every increment of fidelity (responsiveness), aggressive behaviour declines by 0.2448 R square (adjusted) : The treatment intensity of the cognitive behavioural session, duration of service hours and total group responsiveness contributed to 76% of the variability in Aggression levels. Fidelity levels related to group responsiveness are positively correlated with the outcomes measuring changes in aggression levels but contribute less than treatment related factors 19
  20. 20. Step 4 Calculation cont’d Use Multi-Level AnalysisInclude fidelity scores in the ANOVA or regression equationUse a multilevel analysis approach: Group fidelity measure Participant responsiveness Implementation measures Quality of Implementation Individual measures Dosage (quantity)Ensure all key quantifiable fidelity scores are tested for separatecohorts (i.e. various classrooms, sites or session). Correlatefidelity scores with key results (i.e. changes in knowledge,attitudes and behavioural changes) 20
  21. 21. Step 5: Reporting Results What Really Contributed to the Results? High Levels of Fidelity Low levels of Fidelity to the to the Program ProgramFavourable Changes Determine whether fidelity scores or core treatment measures are Made in the Increased contributing to resultsOutcomes of Interest Confidence that the Qualify results by explaining how Program is Effective fidelity may have contributed to (It Works!!) results Be cautious about reporting on program attribution Unfavourable Explore other sources of data to Changes explain findings: Increased Made in the How many elements of the Confidence that the fidelity index were low?Outcomes of Interest Program is Ineffective What elements of the index were low? 21
  22. 22. STEP 5: Determine if Program Outcomes are related to Fidelity (Towards No Drugs sample)Outcome FidelityProgram-specific knowledge 0.33 (0.09)***Beliefs: health-as-a-value 0.10 (0.04)**Beliefs: pro-drug myths -0.15 (0.06)**Cigarette intentions -0.08 (0.04)**Marijuana intentions -0.09 (0.04)**Alcohol intentions -0.07 (0.04)*Multi-level models * p< .10 ** p < .05 *** p < .0001, one tailedRohrbach, L.A., Gunning, G., Sussman, S., & Sun, P. (May; 2008).Predictors of implementation in the Project Towards No Drug Abusedissemination trial. 22
  23. 23. SummaryAttempt to utilize 5 categories: Use Implementation, Dosage,Quality, Participant Responsiveness and Monitoring control groupfidelity categories to ensure the tool/index is comprehensive Utilize a participatory approach to identify the key elements Use a statistical approach (backward or forward selection procedures)Evaluators should advocate for the development and use of aquantifiable fidelity tool to be implemented in the evaluationstudy Encourage the use of interval levels of measurement to each item in the tool. This type of measure can provide specific information about incremental changes in the outcomes of interest Qualitative information should also be collected to explain why the results may be favourable or unfavourable 23
  24. 24. SummaryConstruct a multivariate analysis model that will incorporateelements of the fidelity tool Consider using multilevel models that utilize more than one equation that will take into consideration different levels of data (i.e. Towards No Drugs fidelity tool required a multilevel model for the analysis: schools are nested in communities; pupils are nested in schools etc..)Report how the fidelity levels related to the outcomes of interest Where fidelity levels and expected results are low, ensure that triangulation with other data is used to verify program attribution levels Explore other elements of the regression or other multivariate equations used to isolate what program elements may or may not have contributed to the results. 24
  25. 25. Contact InformationDonna Smith-Moncrieffe, BSc., Crim Dip, MSc. Senior Evaluation Advisor Public Safety Canada National Crime Prevention Center Policy Research and Evaluation Division E-mail: donna.smith-moncrieffe@ps.gc.ca 25

×