Presentation given to What Works Centre for Crime Reduction partners on 25th November 2014 (including research staff from the College of Policing). Introducing the EMMIE evidence appraisal tool.

1. EMMIE training workshopEvidence appraisal in crime prevention

2. Agenda for workshop
12.30 Introduction to EMMIE and rationale of approach (Shane Johnson)
12.45Evidence appraised to date
13.00 The coding instrument
13.30Exercise
14.30The EMMIE narratives (and meta-synthesis)
15.00Finish

3. Profile of (original) reviews

4. Focus of reviews
326 reviews
~125 on single intervention topics
~60 interventions in total
+12 reviews in production

5. Profile of (single intervention) reviews

6. Reviews appraised so far
1.CCTV
2.Lighting
3.Multi-systemic therapy
4.Alcohol ignition interlocks
5.Sobriety checkpoints
6.CPTED (retail robbery)
7.Neighbourhood Watch
8.Music making interventions
9.Electronic monitoring
10.Increased police patrols

7. Experience of applying EMMIE
•Most reviews don’t use the language of EMMIE
–Different fields have very different reporting conventions
•The evidence is generally weak on effect, and often on other dimensions
–But need to remember that reviews rely on primary study evidence
•BUT, weak evidence on effect doesn’t undermine other dimensions
–I.e. reviews can be strong on moderators or implementation
•Appraising quality is subjective, so we automated the scoring

8. Experience of applying EMMIE
Codebook has constantly been challenged and refined
•Effect: meta-analyses conducted in various ways
•Mechanism: presented (or not) in many different ways
•Moderators: working out a priori / post hoc can be challenging
•Implementation: teasing this out from MM tricky sometimes
•Economics: evidence rare on this

9. The coding instrument
•EMMIE-E-relatesto the ‘evidence’ that emerges from reviews
•EMMIE-Q–relates to the ‘quality’ of it
•Both are needed for prospective users to gauge what is (not) known and a level of confidence associated with findings
•Coding instrument appraises both of these for each dimension of EMMIE

10. Exercise
•4 groups
•Identify information on EFFECT and one other dimension
•45 minsreading, discussing and annotating
•15 minutes for group discussion

11. The EMMIE narratives (and meta-synthesis)
•Turning the coding into an accessible format
•Quality assurance
•The narratives

12. Summarising the effect
Rating
Interpretation
XX
Overall, evidence suggests an increase in crime
XX!
Overall, evidence suggests an increase in crime (but some studies suggest a decrease)
XX
Overall , no evidence to suggest an impact on crime (but some studies suggest an increase)
X X
No evidence to suggest an impact on crime
X X
Overall, evidence suggests no impact on crime (but some studies suggest either an increase or a decrease)
X X
Overall, evidence suggests no impact on crime (but some studies suggest a decrease)
X !
Overall, evidence suggests a decrease in crime (but some studies suggest an increase)
X X
Overall, evidence suggests a decrease in crime

13. Summarising EFFECT-Q
Star rating for Effect Q
Associated text
★★★★
The review was sufficiently systematic that mostforms of bias that could influence the study conclusions can be ruled out.
★★★★
The review was sufficiently systematic that manyforms of bias that could influence the study conclusions can be ruled out.
★★★★
Although the review was systematic, someforms of bias that could influence the study conclusions remain.
★★★★
Although the review was systematic, manyforms of bias that could influence the study conclusions remain.
★★★★
Text to reflect specific review

14. Star rating for Moderator Q
Associated text
★★★★
Collection and analysis of relevant data relating to theoretically grounded moderators and contexts
★★★★
Theoretically grounded description of relevant contextual conditions
★★★★
Tests of the effects of contextual conditions defined post hoc using variables that are at hand
★★★★
Ad hoc description of possible relevant contextual conditions
★★★★
No reference to relevant contextual conditions that may be necessary
Summarising MODERATOR-Q

15. Meta-synthesis
•Synthesis methods needed for integrating reviews
•Two overarching decision rules for narratives:
1.For each EMMIE element use the HIGHEST QUALITY (highest Q score) scoring review to populate the EMMIE-E and EMMIE-Q scores
2.For each piece of information make it clear which source is being referred to
•Use sub-group analysis FROM ALL REVIEWS to work out the ‘inside’ cross and tick
–i.e. evidence of statistical reduction or backfire under certain conditions)
–Use the HIGHEST QUALITY meta-analysis to populate the overall effect

17. Thanks everyone