Uploaded byNick Hart, Ph.D.
Improving Environmental Evidence in the US
The document discusses evaluation capacity at the U.S. Environmental Protection Agency (EPA), highlighting barriers and facilitators to conducting effective evaluations within its programs. Despite the EPA's extensive resources and regulatory framework, it performs limited evaluations compared to the number of rulemakings. The paper suggests strategies for improving evaluation capacity, emphasizing the need for learning processes and broader infrastructure to better support evaluative practices.
More Related Content
Similar to Improving Environmental Evidence in the US
Improving Environmental Evidence in the US
- 1. 1 Improving Environmental Evidence: Barriersand Facilitators of Evaluation Capacity at the U.S. Environmental Protection Agency Nicholas R. Hart, PhD @NickRHart January 9, 2017 Centre for the Evaluation of Complexity Across the Nexus (@cecanexus) United Kingdom THE GEORGE WASHINGTON UNIVERSITY WASHINGTON, DC
- 2. USEPA Programs andRegulated Entities • Mission to protect human health and the environment • Implement suite of Federal environmental statutes with emphasis on air, land, water, and chemicals through Federal-State partnership • 100+ programs (as defined by projects in the EPA budget) – $9 billion per year in appropriated funds, plus $7 billion in state funds • 800,000+ regulated facilities in U.S. – Environmental regulations est. to cost ~$330 billion annually 2Improving Environmental Evidence@nickrhart
- 3. Evidence Status Quoat USEPA • EPA conducts extensive ex ante or prospective analysis during rulemakings (~1,650 over past 15 years) – Cost Benefit Analysis (regulatory impact assessments) – Health Risk Assessments • A robust scientific research network to support regulatory actions • Relatively little ex media or ex post (or retrospective) evaluation – EPA sponsored about 70 evaluations over the past 15 years, including 3 impact evaluations – Evaluated programs often not the same as those with prospective analyses 3Improving Environmental Evidence@nickrhart
- 4. Example: Saccharin Regulations •In 1980 EPA listed as a hazardous waste based on available scientific literature – Determination supported by literature on rats, with effects extrapolated to humans • In 2000, toxicology community identified that the studies on which EPA’s decision had relied on studies with rats the identified biological mechanisms not in humans – A public request led EPA to re-consider the information underlying the listing • In 2010, EPA de-listed saccharin as a hazardous waste 4Improving Environmental Evidence@nickrhart
- 5. Overview of EvaluationCapacity at EPA • Limited capacity at EPA today to conduct evaluation, notwithstanding a strong performance monitoring culture and emphasis on process studies • Question: Why the gap in capacity? – Specifically, what are the barriers and facilitators of production? • Isn’t this just a function of mandates? Answer is much more complicated. – It’s true that there is no existing agency-wide mandate for USEPA to conduct program evaluations, but some programs do have mandates – When regulatory reviews are required by law, rarely completed 5Improving Environmental Evidence@nickrhart
- 6. Evaluation Capacity Literature •What is evaluation capacity building? – “Evaluation capacity building (ECB) is an intentional process to increase individual motivation, knowledge, and skills, and to enhance a group or organization’s ability to conduct or use evaluation.” (Labin et al. 2012) • Over past 20 years, the ECB field developed numerous case studies of organizational capacity – more recent efforts to generalize in evaluation capacity frameworks or to develop tools to gauge capacity • But there is still value in using case studies in this field -- perhaps even a need to return to case study approach for studying government capacity – Existing literature largely based on small NGOs, not large institutions – Regulatory nature of EPA potentially substantially different context 6Improving Environmental Evidence@nickrhart
- 7. Case Study Approach •Case Studies of USEPA programs: hazardous waste, air quality, and performance partnerships – EPA is a large, diverse agency; case studies allowed for understanding programs and cultures across agency – Allows for in-depth consideration of processes, factors, and implications of evaluation with specific EPA programs – Included one program with extensive evaluation experience, and two programs with more limited experience • Data: –57 semi-structured interviews with state and Federal officials, and other stakeholders –Every evaluation completed by respective programs, as well as relevant laws and evaluation policies 7Improving Environmental Evidence@nickrhart
- 8. Evaluation Capacity Factorsat USEPA • 10 key factors, observed across each of the three cases, each comprised of a cluster of multiple themes and nodes • Factors typically identified as both barriers and facilitators – Evaluation capacity appears to occur on a spectrum rather than in a dichotomy – Barriers observed to limit activities, whereas facilitators come together with “evaluation window” for production to occur – Some barriers will almost always close the window and some barriers more difficult to overcome than others 8Improving Environmental Evidence@nickrhart
- 9. Evaluation Capacity Factorsat USEPA Resources Methods Data 9Improving Environmental Evidence@nickrhart
- 10. Context Matters 10Improving EnvironmentalEvidence@nickrhart
- 11. Evaluation Capacity Factorsat USEPA Resources Methods Data Cultural Context Political Context 11Improving Environmental Evidence@nickrhart
- 12. Context Matters: Litigation 12ImprovingEnvironmental Evidence@nickrhart
- 13. Context Matters: LegislativeAttention 13Improving Environmental Evidence@nickrhart
- 14. Context Matters: Training 14ImprovingEnvironmental Evidence@nickrhart
- 15. Context Matters: Appointees 15ImprovingEnvironmental Evidence@nickrhart
- 16. Evaluation Capacity Factorsat USEPA Perceived Utility DesignMotivation Leadership Resources Methods Data Legal Cultural Context Political Context 16Improving Environmental Evidence@nickrhart
- 17.
- 18. Implications For Practiceat USEPA • Evaluation is possible at EPA, even though not currently widespread • Demonstrated success stories exist in all three case studies, though level of program capacity varied • No one-size-fits-all design identified for establishing capacity in EPA programs • Some factors may be more productively targeted at EPA to improve capacity moving forward, in particular in “Tier 1” – traditional factors emphasized in government activities may not be the best place to start 18Improving Environmental Evidence@nickrhart
- 19. 1. Utilize Centralizedand Decentralized Capacity 2. Identify Gaps Using Capacity Inventories 3. Incentivize Evaluation Within Existing Processes 4. Employ Participatory Evaluation to Align with Existing Collaborative Practices 5. Learn by Doing A Path Forward for Improving USEPA’s Evaluation Capacity 19Improving Environmental Evidence@nickrhart
- 20. • With prospectiveanalyses, emphasis on meta-analyses in toxicological, epidemiological, and observational research will likely continue • For policy reviews and evaluation – EPA needs learning processes in place to improve. Just as prospective analyses today are imperfect, a widely accepted view among interviewees in this research, so too will be some evaluations – Must overcome aversions from environmental interest groups to evaluation in the US EPA • Much work to do before larger systematic reviews become routine Final Word on Improving Environmental Evidence in the U.S. 20Improving Environmental Evidence@nickrhart
- 21. Nick Hart George WashingtonUniversity Washington, D.C. nick.r.hart@gmail.com @NickRHart Thank You! 21Improving Environmental Evidence@nickrhart
- 22. Case Study Approach 22ImprovingEnvironmental Evidence@nickrhart
- 23. Relative Importance ofthe Factors 23Improving Environmental Evidence@nickrhart
Editor's Notes
- #2 Good afternoon and thank you for joining today. And Peter – thank you for the invitation to speak. I’d like to present some recent research that I conducted that considers our ability to produce evidence and program evaluations at the institutional level. USEPA is a different type of agency from where much of our existing research on evaluation capacity in organizations has occurred. I’ll highlight some of those important features that make US EPA distinct, then talk a bit about what I found in studying U S EPA’s evidence enterprise at an institutional level, And leave you with conclusions at the end of the discussion about a potential path forward for building stronger body policy evidence at EPA, that may be transferable to other regulatory or institutional settings.
- #3 First, a bit of background about US EPA. USEPA is one of the premiere environmental regulatory bodies in the world. In the US, we task the EPA with protecting human health and the environment through a suite of federal laws – covering all media: air land water. EPA considers that they have over 100 individual programs – though really many more – and these programs are jointly implemented by the Federal government and each of the 50 states (as well as tribes and territories) More than 800,000 individual entities regulated under the laws and estimated cost in the hundreds of billions of US dollars each year So the weight of the programs is practically and economically substantial – therefore changes in these programs can carry a lot of weight
- #4 The evidence status quo at EPA is really interesting. A vast amount of information is utilized to guide policy decisions at the outset From 1998-2013 ~200 RIAs (cost-benefit analyses) and ~1450 Ecological and Human Health Risk Assessments These types of analyses are built on high quality information, but often draw many assumptions in probabilistic models – to predict likely impacts of policies over time But relatively little effort is made to analyze those same regulations and policies over time with a formal evaluation infrastructure
- #5 EPA had no formal operational process in place to review the decision – that’s why it took 40 years to review a decision that was in fact an error We can – EPA can do better, but doing so requires a diligent process and attention to reviewing decisions and activities continuously The institutional level of activities around building, reviewing and using evidence are all really important Today, this isn’t really done
- #6 The key question I set out to answer – why is there a gap in evaluation capacity relative to prospective analysis? Some might argue its just because one type of review is required and the other is not – but not that simple
- #7 Why evaluate? Accountability, learning, inform decision-making Over Past 20 years – start with cases, then frameworks, then tools to measure – but skipped a lot of detail. Today, not clear the tools are even measuring the right constructs. Still value in focusing on the cases, because of distinctions in types of capacity and types of organizations.
- #8 Case Selection Criteria: Implement national-scale EPA policy Well-established programs in operating stage Joint implementation between EPA and state partners What was considered an evaluation?? DEFINE EPA produced or supported Social science method
- #9 Coming out of the three cases were 10 key factors, identified in each of the cases Perhaps not surprisingly the factors that affected evaluation capacity for production were along a spectrum – not purely dichotomous But when the factors tended to lean toward predominantly barriers, then activities were limited When leaned toward facilitators, with enough of them, activities were more likely One key element of this relationship was that when enough facilitators came together the “evaluation window” seemed to quickly allow for evaluation production But certain barriers could almost always close that window, presenting challenges for the program
- #10 First, the technical issues: resources – methods – data This is what literature tends to emphasize as important for capacity – the technical attributes But not really the case at an institutional level – these are present, but not the most crticial for EPA Yes, need money and people. But if a senior manager decides its important, they will find a way. Similarly, rarely ask if data are available at outset, decide to conduct an evaluation then proceed rather than the other way around Second , as I mentioned context matters. Especially political and cultural. All of the evidence-building activities are interacting iteratively with this context.
- #11 I come from the Lee Cronbach mindset when we talk about evaluation – he said “context matters!” There is a lot of important context around the issues I will discuss, and I will try to bring in a few examples First, on Money. Often one of the first issues raised in interviews. But it’s not the most important, just the easiest to describe. Here in this chart you can see EPA resources have actually been relatively stable over time, with few punctuations of change Static resources alone don’t mean there’s not money – how money is spent is an issue of prioritization But this is my way to say, that the answer to the challenges about evaluation capacity isn’t just about money
- #12 The Second group of factors are contextual -- as I mentioned context matters. Especially political and cultural. All of the evidence-building activities are interacting iteratively with this context.
- #13 In recent history much of EPA’s work has been politically contentious. Sometimes this manifests because there are clear winners and losers in many regulatory actions. One consequence is of course litigation – perceived losers often file lawsuits challenging actions. In the 20 year window there were more than 100 lawsuits filed against EPA each year. And mixed by private sector and environmental groups. Sometimes even local governments! Many of these lawsuits lead to attention in the executive and legislative political discourse – drawing attention to the agency’s actions
- #14 At the same time, congressional attention to EPA is not a constant. In 2013, as a share of total congressional action, there were fewer environmental hearings than any time since the 1960s. So When an agency’s actions are politically contentious, there are some incentives to try to keep your head down, and to stay out of the spotlight if at all possible.
- #15 Then of course, the backgrounds of people at EPA affect the cultural context of the agency – not to mention the skillsets. 2/3 of EPA’s white collar staff are made up of hard scientists, lawyers, engineers. While these statistics are a little dated, This chart shows the breakout of those with advanced degrees. There are relatively few experts on program evaluation hired by the agency, and few social scientists calling for these types of analyses in the culture.
- #16 One component of that culture – most of the politically appointed administrators of EPA have come from graduate training in law, engineering, or hard sciences! So you can imagine what this does for the decision-making apparatus within the organization
- #17 This brings us to our third group of factors – that deal with senior leaders: leadership and motivation We have been saying for 20 years that leadership is important – and it is. But we often don’t focus on the thing that comes before: motivation at an institutional level (aggregations of individual motivations) Without motivation to consider or pursue evaluation, program managers and staff may never even consider the other factors that affect supply. Reasons often included: accountability, responses to program critiques, implementation awareness and a desire to demonstrate what capable of achieving, and demand instigated by government-wide initiatives. Leadership: attributes of individual people – people in positions of authority – and this reflects their priorities. Low prioritization of evaluation was consistently identified as a barrier for producing evaluation at USEPA. This was even the case in 2013 when the EPA Adminsitrator effectively termianted the central office at the agency, favoring business process improvements over systematic evaluation. In this instance – leaders de-prioritized evaluation almost completely. Finally, the last cluster of factors are the ones that can place a “full stop” on the production of evaluation. They tend to lean toward being barriers – and were often strong enough to close open evaluation windows Design – Program design includes far more information than what is included in statute. Design also reflects how the executive branch, and stakeholders, interpret the law when designing the program. Includes: "over-emphasis on institutionalizing a process" and following SOPs -- as a major impediment when the activity of evaluation was itself not institutionalized Legal – EPA is often under pressure to meet new requirements legally imposed on the agency. Many citizen suits result in settlement agreements, or court-ordered timelines, for the agency to take some action. Evaluations that jeopardized the evidence on which the case for a regulation was made might invite lawsuits – “fear of instigating new legal challenges.” Utility -- Beliefs about the value of program evaluation
- #18 this framework varies from existing research by suggesting that capacity not only operates within a contextual dimension, but also that some aspects of evaluation can be so negative as to stop capacity for supply in particular areas altogether. One final observation regarding the visualization of factors presented in Figure 8.4 is that the two-dimensional conceptualization may actually be insufficient to understand capacity within EPA, that is a complex and hierarchical organization. For example, in a three dimensional version of Figure 8.4, the model might be stratified by headquarters, regions, states, or even offices within each geographic component. 3D model – incredibly complicated, only scratching surface here
- #21 Lots of great work going on in evidence building today at EPA But evaluation activities are very weak – much work to do here to strengthen the capabilities for imporving the agency over time
- #22 Thank you for listening and I look forward to your questions
- #23 Research starts by asking can evaluation be done – building an inventory Explain Joe Wholey work on evaluability assessment: The point here is that the more yes’s the stronger the case for evaluation, but don’t necessarily need yes in each