This document discusses evaluation research and problem analysis in criminal justice. It defines evaluation research as seeking to evaluate the impact of interventions and problem analysis as helping officials choose alternative actions. It describes the policy process and how evaluation research links intended actions and goals to empirical evidence. Different research designs for evaluation are discussed, including randomized experiments and quasi-experimental designs. Problem analysis uses the same research methods as evaluation to analyze problems and responses. The political context of applied research and managing stakeholder interests are also covered.

1. 1
Chapter 13:
Evaluation
Research and
Problem Analysis
© 2018 Cengage Learning. All Rights Reserved.

2. 2
© 2018 Cengage Learning. All Rights Reserved.
Learning Objectives
• Summarize evaluation research and problem analysis as
examples of applied research in criminal justice
• Describe how different types of evaluation activities
correspond to different stages in the policy process
• Explain the role of an evaluability assessment
• Understand why a careful formulation of the problem,
relevant measurements, and criteria of success or failure
are essential in evaluation research
• Describe the parallels between evaluation research designs
and other designs
• Explain the advantages, requirements, and limits of
randomized field experiments

3. 3
© 2018 Cengage Learning. All Rights Reserved.
Learning Objectives, cont.
• Summarize the importance of process evaluations conducted
independently or in connection with an impact assessment
• Describe the role of problem analysis as a planning technique
that draws on the same social science research methods used
in program evaluation
• Explain how the scientific realist approach focuses on
mechanisms in context, rather than generalizable causal
processes
• Present an example of how criminal justice agencies are
increasingly using problem analysis tools, crime mapping, and
other space-based procedures
• Explain how evaluation research entails special logistical,
ethical, and political problems

4. 4
© 2018 Cengage Learning. All Rights Reserved.
Introduction
• Evaluation Research: Refers to a research
purpose rather than a specific method; seeks to
evaluate the impact of interventions; if some result
was produced
• Problem Analysis: Designed to help public officials
choose from alternative future actions
• Policy Intervention: An action taken for the
purpose of producing some intended result
• Evidence-Based Policy: The actions of justice
agencies are linked to evidence used for planning
and evaluation

5. 5
© 2018 Cengage Learning. All Rights Reserved.
The Policy Process
• Begins with a demand supporting a new course
of action or opposition to existing policy
• Policymakers consider ultimate goals and actions
to achieve those goals
• Outputs: The means to achieve desired goals
• Impacts: Refer to basic questions about what a
policy seeks to achieve
• If some policy is taken, then we expect some
result

6. 6
© 2018 Cengage Learning. All Rights Reserved.
Discussion Question 1
What if you studied policy implementation?
Do you think you’d be more interested in
impacts or outputs? Why?

7. 7
© 2018 Cengage Learning. All Rights Reserved.
Linking the Process to Evaluation
• Are policies being implemented as planned?
• Are policies achieving their intended goals?
• Evaluation seeks to link intended actions and
goals of policy to empirical evidence that:
• Policies are being carried out as planned (process evaluation)
• Policies are having the desired effects (impact assessment)
• Often conducted together

8. 8
© 2018 Cengage Learning. All Rights Reserved.
Getting Started
• Learning policy goals is a key first step in doing
evaluation research
• Evaluability Assessment: “Pre-evaluation”—
researcher determines whether requisite
conditions are present
• Support from relevant organizations
• What goals and objectives are; how they are translated into
program components
• What kinds of records or data are available
• Who has a direct or indirect stake in the program

9. 9
© 2018 Cengage Learning. All Rights Reserved.
Discussion Question 2
How might you convince a public official that
an evaluation is both a necessary thing and
a positive thing?

10. 10
© 2018 Cengage Learning. All Rights Reserved.
Problem Formulation and Measurement
• Different stakeholders often have different goals and
views as to how a program should actually operate
• Must clearly specify outcomes—program goals and
objectives
• Create objectives—operationalized statements
• Definition and measurement: target/beneficiary
population; decide between using current measures
or creating new ones
• Measure program contexts, outcomes, and delivery

11. 11
© 2018 Cengage Learning. All Rights Reserved.
Designs for Program Evaluation
• Randomized Evaluation Designs: Avoids selection
bias; allows assumption that groups created by
random assignment are statistically equivalent;
may not be suitable when agency or staff makes
exceptions
• Case Flow: Represents process through which
subjects are accumulated into experimental and
control groups
• Treatment Integrity: Whether an experimental
intervention is delivered as intended; ≈ reliability
• Threatened by midstream changes in program

12. 12
© 2018 Cengage Learning. All Rights Reserved.
Discussion Question 3
What if local researchers wanted to
experiment on the criminal justice system in
your area? What would be your concerns as
far as ethics and integrity?

13. 13
© 2018 Cengage Learning. All Rights Reserved.
Conditions for Randomized Experiments
• Staff must accept random assignment and agree to
minimize exceptions to randomization
• Case flow must be adequate to produce enough
subjects in each group so that statistical tests will be
able to detect significant differences in outcome
measures
• Experimental interventions must be consistently
applied to treatment groups and withheld from
control groups
• Need equivalence prior to intervention, and ability to
detect differences in outcome measures after
intervention

14. 14
© 2018 Cengage Learning. All Rights Reserved.
Home Detention: Two Randomized Studies
• Combining home detention with ELMO
• Juvenile program paid less attention to
delivering program elements and using ELMO
info than adult
• Difficult to maintain desired level of control over experimental
conditions
• Also difficult when more than one organization is involved
• Randomization does not control for variation in
treatment integrity and program delivery; utilize
other methods

15. 15
© 2018 Cengage Learning. All Rights Reserved.
Quasi-Experimental Designs
• No random assignment to Experimental and
Control group
• Often “nested” in experimental designs as backups
• Lack built-in controls for selection and other
Internal Validity threats
• You must construct Experimental and Control
groups that are as similar as possible

16. 16
© 2018 Cengage Learning. All Rights Reserved.
Quasi-Experimental Designs, cont.
• Ex post Evaluation: Conducted after experimental
program has gone into effect
• Full Coverage Programs: Sentencing guidelines
• Larger Treatment Units: Neighborhood crime
prevention program
• Interrupted Time-Series Designs: Require
attention to different issues because researchers
cannot normally control how reliably the
experimental treatment is actually implemented
• Instrumentation, History, Construct Validity

17. 17
© 2018 Cengage Learning. All Rights Reserved.
Problem Analysis and Scientific Realism
• Problem analysis, coupled with scientific realism, helps
public officials use research to select and assess
alternative courses of action
• Realists suggest that similar interventions will have
different outcomes in different contexts
• Evaluators should search for mechanisms (IVs) acting
in context (assorted intervening variables) to explain
outcomes (DVs)
• Appropriate in small-scale evaluations directed toward
solving a particular problem in a specific context

18. 18
© 2018 Cengage Learning. All Rights Reserved.
Problem Analysis
• Problem-Oriented Policing
• Problem solving: A fundamental tool in problem-oriented
policing
• How-to-Do-It Guides: A general guide to
crime analysis to support problem-oriented
policing
• Problem & Response Guides: Describe how
to analyze very specific types of problems
and what are known to be effective or
ineffective responses

19. 19
© 2018 Cengage Learning. All Rights Reserved.
Auto Theft in Chula Vista
• Nanci Plouffe and Rana Sampson (2004)
began their analysis of vehicle theft by
comparing Chula Vista to other southern
California cities
• Theft rates tended to be higher for cities closer to the
border
• Ten parking lots accounted for 25 percent of thefts and
20 percent of break-ins in the city
• Six of the ten lots were among the top ten calls-for-
service locations in Chula Vista
• Auto theft hot spots also tended to be hot spots for other
kinds of incidents

20. 20
© 2018 Cengage Learning. All Rights Reserved.
Other Applications of Policy Analysis
• Space- and Time-Based Analysis: increased
prevalence due to technological advances
• Crime maps usually represent at least four
different things:
• (1) one or more crime types; (2) space or area; (3) some
time period; and (4) some dimension of land use, usually
streets
• Problem-solving tools and processes
• Strategic Approaches to Community Safety Initiatives
(SACSI)

21. 21
© 2018 Cengage Learning. All Rights Reserved.
Political Context of Applied Research
• Different stakeholder interests can produce
conflicting perspectives on evaluations
• Researcher must identify stakeholders and
perspectives
• Educate stakeholders on why evaluation should
be conducted
• Explain that applied research is used to
determine what works and what does not
• Political concerns and ideology may color
evaluation; be careful