Insights from Program Evaluation for Retrospective Reviews of regulations -- Presented at the Society for Risk Analysis on December 8, 2015

1. 1
Insights from Program Evaluation
for Retrospective Evaluation of Regulations
Nicholas R. Hart
@nickrhart
PhD Candidate, Trachtenberg School of Public Policy and Public Administration
December 8, 2015
Society for Risk Analysis
Arlington, VA
THE GEORGE WASHINGTON UNIVERSITY
WASHINGTON, DC

2. Setting the Stage…
2Hart Evaluation Insights for Retrospective Reg Evaluation
• Program Evaluation: application of systematic analytical (social
science) methods to address questions about program
operations and results; includes:
– Performance Measurement & Monitoring: routine measurement of
program inputs, outputs, intermediate/ long-term outcomes attributed
to a program
• Both involve measurement + judgment

3. • Program Evaluation refers to both a mindset and the
application of analytical skills, and is a profession.
• 3 especially relevant elements of evaluation practice for
evaluating regulatory impact:
1. Articulate the Theory of Change
2. Frame Appropriate Evaluation Questions
3. Match Design to Evaluation Questions
Applying Program Evaluation to Regs
3Hart Evaluation Insights for Retrospective Reg Evaluation

4. Key Components
• Conceptual: explicit theory or model of how the program/policy is
expected to cause the intended outcomes
• Empirical: evaluation guided by the model.
Modeling the Theory of Change
• What elements constitute the “treatment (regulation)”?
• What are the desired outcomes – in the longer run and in the
shorter run?
• What is the expected process to produce desired outcomes?
• How are the program elements supposed to relate to one another?
• What are important bridging assumptions about how processes and
elements are supposed to relate to on another?
• What are contextual (or mediating) factors outside of the control of
program implementers that may affect the ability of the program or
policy to produce the desired outcomes?
#1: Articulate a Program or Policy’s
Theory of Change
4Hart Evaluation Insights for Retrospective Reg Evaluation

5. Basic Theory of Change
5Hart Evaluation Insights for Retrospective Reg Evaluation
Program
Elements
Mediators Outcomes
Moderators

6. Generic Logic Model
6Hart Evaluation Insights for Retrospective Reg Evaluation
Inputs
People
Money
Clients
Case-load
Processes
Activities
Initiatives
Procedures
Outputs
Services
Products
Outcomes
Impacts
Effects
Results
Intervening (Contextual) Variables

7. 7Evaluation Insights for Retrospective Reg ReviewsHart
Regulations
State
Administrative
Capacity
Number of
Permits
Reviewed
State Political
Will
State and
Federal
Enforcement
Public Funding
Enforcement
Intensity
Decreased
Absentee-
ism
Lower Levels
of Health
Problems
Higher
Educational
Achievement
Improved
Quality of Life
Environmental
Protection
Non-profit
Organization
Capacity
Lower Emissions
Lower Exposure to
Ambient Air
Pollution (Lead)
Lower
Levels of
Learning
Defects
Family
Education
and Income
Inputs
Mediating
Factors* Outputs
Short-Term
Outcomes
(~1 year)
Mediating
Factors*
Intermediate
Outcomes
(1-3 years)
Long Term
Outcomes
(4 + years)
Assessing the Impact of Environmental Regulations on Lead Emissions
Quality of
Education
(School
Level)
Geographic
and
Meteorological
Conditions
Non-Illness
Related
School
Absences
Individual
Test Taking
Skills
Reduced
Neurological Effects
Higher IQs
Higher Test
Scores
Longer Life
Expectancy* Mediating Factors are factors outside of the control of
government officials which, given variations in levels of
their intensity and scope, may positively or negatively
affect achievement of desired outcomes.
Reduced Blood
Lead Levels
Lower Ambient
Pollution
Concentrations
(Lead)

8. • Desired outcomes may be externally established
(e.g., by Congress)
• Increased complexity from intergovernmental
relations in implementation and enforcement (e.g.,
Federal, State, Local levels)
• Place requirements on private actors to achieve public
goals (compared to direct provision of services or programs)
• Behavioral responses to requirements affect outcomes
• Often national scope, relying on ex ante analysis with
focus on averages (instead of diversity of local conditions and
preferences)
Consider Distinctive Features of Regulation
8Hart Evaluation Insights for Retrospective Reg Evaluation

9. • Assumptions about Linkages -- e.g., reduced U.S. vehicle
emissions will reduce GHG concentrations, which will
reduce effects of anthropogenic climate change
• Behavioral Responses -- e.g., manufacturers meet standard
with smaller and/or alternate fuel vehicles; drivers drive
more miles when cost-per-mile is lower
• “Unobservables” -- e.g., consumers’ preferences for
different vehicle attributes
• External Forces -- e.g., consumer behavior affected by fuel
prices and economic conditions. GHG concentrations
affected by other countries’ emission patterns
Identify Key Contextual Factors
9Hart Evaluation Insights for Retrospective Reg Evaluation

10. • The type of question – and type of desired evaluation –
matter for determining appropriate methods:
– Descriptive Questions – describe input levels, outputs,
contextual variables and/or measurable outcomes
– Normative Questions – assess levels of compliance for
outputs or outcomes with criteria in law, regulation, or equity
norms
– Impact Questions – measures of effectiveness or “impact”
– Explanatory Questions – explain implementation (fidelity)
and processes
#2: Frame Useful Evaluation Questions
10Hart Evaluation Insights for Retrospective Reg Evaluation

11. Types of Evaluation
11Hart Evaluation Insights for Retrospective Reg Evaluation
Formative (Process/Implementation) Evaluation
Outcome
Evaluation
Impact
Evaluation
Desired
Impact
Long-term
Outcomes
Intermediate
Outcomes
Short-term
Outcomes
OutputsActivities
Resources
& Inputs
WHYHOW
Focus of Evaluation
(Newcomer)

12. Principles of Evaluation Design
1. Frame the most appropriate questions to address
2. Clear and answerable evaluation questions drive the design
decisions
3. Design decisions are made to provide appropriate data and
comparisons needed to address the evaluation questions
4. Decisions about measurement and design are made to bolster
the methodological integrity of the results
5. During design deliberations careful consideration should be
given to strengthening inferences about findings
6. Goal to report that evaluation design and reporting decisions
were characterized by strong methodological integrity
#3: Match Design to Research Questions
12Hart Evaluation Insights for Retrospective Reg Evaluation

13. Examples
Objective Illustrative Questions Possible Design
#1: Describe
program
(or regulatory)
activities
 How extensive and costly are the regulatory enforcement
activities?
 How do implementation efforts vary across sites,
beneficiaries, regions?
 Has the regulation been implemented sufficiently to be
evaluated?
 Performance Measurement
 Exploratory Evaluations
 Evaluability Assessments
 Multiple Case Studies
#2: Probe
targeting &
implementation
 How closely are the protocols implemented with fidelity
to the original design?
 What key contextual factors are likely to affect intended
outcomes?
 What feasibility or management challenges hinder
successful implementation of the regulation?
 Multiple Case Studies
 Implementation or Process
evaluations
 Performance Audits
 Compliance Audits
#3: Measure
impact of policy
or regulation
 What are the average effects across different
implementations of the regulation?
 Has implementation of the regulation produced results
consistent with its design (espoused purpose)?
 Is the implementation strategy more (or less) effective in
relation to its costs?
 Experimental Designs/RCTs
 Non-experimental Designs:
Difference-in-difference,
Propensity score matching, etc.
 Cost-effectiveness & Benefit
Cost Analysis
 Systematic Reviews & Meta-
Analyses
#4 : Explain how/
why produce
(un)intended
effects
 How/why did the regulation have the intended effects?
 To what extent has implementation of the regulation had
important unanticipated negative spillover effects?
 How likely is it that the regulation will have similar effects
in the future?
 Impact Pathways and
Process tracing
 System dynamics
 Configurational analysis, etc.

14. • There are and will continue to be practical challenges and
impediments for regulatory evaluation:
– Lack of incentives
– Methodological challenges
• But regulations can be appropriately designed to overcome the
impediments to enable ex post program/policy evaluation
• Ultimately more attention to ex post regulatory evaluation will improve
the regulatory environment by:
– Helping to identify underperforming regulations
– Improving future regulations (and ex ante analysis)
• Program evaluation thinking and tools can be productively applied to
regulations
Conclusions
14Hart Evaluation Insights for Retrospective Reg Evaluation

15. Thank you!
Nick Hart
nrhart@gwu.edu
@nickrhart
www.nickhart.us
15Evaluation Insights for Retrospective Reg ReviewsHart

16. Appendix
16Evaluation Insights for Retrospective Reg ReviewsHart

17. Matching Designs to
the Evaluation Questions
Evaluation
Objective
Illustrative Questions Possible Design
#1 Describe
program (or
regulatory)
activities
 Who does the regulation affect –
both targeted organizations and
affected populations?
 What activities are needed to
implement the regulation? By
whom?
 How extensive and costly are the
regulatory enforcement
activities?
 How do implementation efforts
vary across delivery sites,
subgroups of beneficiaries,
and/or across geographical
regions?
 Has the regulation been
implemented sufficiently to be
evaluated?
 Performance Measurement
 Exploratory Evaluations
 Evaluability Assessments
 Multiple Case Studies
17

18. Matching Designs to
the Evaluation Questions, cont.
Evaluation
Objective
Illustrative Questions Possible Design
#2 Probe
regulatory
implementation
and
targeting
 To what extent has the regulation
been implemented?
 When evidence-based regulations
are adopted, how closely are the
protocols implemented with fidelity
to the original design?
 What key contextual factors are likely
to affect the ability of the regulatory
implementers to have the intended
outcomes?
 What feasibility or management
challenges hinder successful
implementation of the regulation?
 To what extent have activities
undertaken affected the populations
or organizations targeted by the
regulation?
 To what extent are implementation
efforts in compliance with the law
and other pertinent regulations?
 To what extent does current
regulatory targeting leave significant
needs (problems) not addressed?
 Multiple Case Studies
 Implementation or Process
evaluations
 Performance Audits
 Compliance Audits
18

19. Matching Designs to
the Evaluation Questions, cont.
Evaluation
Objective
Illustrative Questions Possible Design
#3 Measure
regulatory
impact
 Has implementation of the
regulation produced results
consistent with its design (espoused
purpose)?
 How have measured effects varied
across implementation approaches,
organizations, and/or jurisdictions?
 For which targeted populations has
the regulation consistently failed to
show intended impact?
 Is the implementation strategy
more (or less) effective in relation
to its costs?
 Is the implementation strategy
more cost effective than other
implementation strategies also
addressing the same problem?
 What are the average effects across
different implementations of the
regulation?
 Experimental Designs, i.e.,
Random Control Trials
(RCTs)
 Difference-in-difference
designs
 Propensity score matching
(PSM)
 Statistical adjustments with
Regression Estimates of
Effects
 Multiple Time Series
Designs
 Regression discontinuity
designs
 Cost-effectiveness Studies
 Benefit Cost Analysis
 Systematic Reviews
 Meta-Analyses
19

20. Matching Designs to
the Evaluation Questions, cont.
Evaluation
Objective
Illustrative Questions Possible Design
#4 Explain how
and why
programs
produce
intended and
unintended
effects
 How and why did the regulation have
the intended effects?
 Under what circumstances did the
regulation produce the desired effects?
 To what extent has implementation of
the regulation had important
unanticipated negative spillover
effects?
 What are unanticipated positive effects
of the regulation that emerge over
time, given the complex web of
interactions between the regulation
and other rules or policies, and who
benefits?
 For whom (which targeted
organizations and/or populations) is
the regulation more likely to produce
the desired effects?
 What is the likely impact trajectory of
the regulation (over time)?
 How likely is it that the regulation will
have similar effects in other contexts
(beyond the context studied)?
 How likely is it that the regulation will
have similar effects in the future?
 Impact Pathways and
Process tracing
 Contribution Analysis
 Non-linear modeling,
system dynamics
 Configurational analysis,
e.g., Qualitative Case
Analysis (QCA)
 Realist-based synthesis
20