Recommended
More Related Content
What's hot
What's hot (20)
Similar to Chapter 4 policy evaluation (dye)
Similar to Chapter 4 policy evaluation (dye) (20)
Recently uploaded
Recently uploaded (20)
Chapter 4 policy evaluation (dye)
- 1. Understanding Public Policy Fifteenth edition Chapter 4 Policy Evaluation Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 2. Modules (1 of 2) • Introduction: Policy Evaluation • 4.1: Policy Evaluation: Assessing the Impact of Public Policy • 4.2: The Symbolic Impact of Policy • 4.3: Program Evaluation: What Governments Usually Do • 4.4: Program Evaluation: What Governments Can Do • 4.5: Experimental Policy Research • 4.6: Federal Evaluation: The Office of Management and Budget Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 3. Modules (2 of 2) • 4.7: Federal Evaluation: The General Accountability Office • 4.8: Program Evaluation: Why It Fails So Often • 4.9: How Bureaucrats Explain Negative Findings • 4.10: Why Government Programs Are Seldom Terminated • 4.11: Politics as a Substitute for Analysis • 4.12: The Limits of Public Policy • Summary: Policy Evaluation Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 4. Learning Objectives (1 of 3) • 4.1: Assess policy impact with respect to its costs and benefits • 4.2: Identify symbolic impact of a policy as the people's perceptions of government action and attitude • 4.3: Review some of the ways in which government agencies find out the effectiveness of their programs • 4.4: Recall the process of systematic program evaluation • 4.5: Analyze arguments against government-sponsored experimental policy research Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 5. Learning Objectives (2 of 3) • 4.6: Outline the responsibilities of the Office of Management and Budget • 4.7: Report the responsibilities of the General Accountability Office • 4.8: Review some of the problems plaguing policy evaluation • 4.9: State some of the ways in which bureaucrats try to defend their policy • 4.10: Analyze reasons for the continuation of inefficient government programs and policies Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 6. Learning Objectives (3 of 3) • 4.11: Identify steps of a political approach to policy analysis • 4.12: Evaluate limitations of policies in addressing all problems Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 7. Introduction: Policy Evaluation • Areas of interest for the government • Issues the government is negligent in Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 8. 4.1: Policy Evaluation: Assessing the Impact of Public Policy (1 of 4) Objective: Assess policy impact with respect to its costs and benefits – Overview of policy evaluation – Policy impact Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 9. 4.1: Policy Evaluation: Assessing the Impact of Public Policy (2 of 4) Objective: Assess policy impact with respect to its costs and benefits • 4.1.1: Measuring Impact, Not Output – Definition of policy output – Difference between policy impact and policy output • 4.1.2: Target Groups – Overview – Various effects of a program on target groups Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 10. 4.1: Policy Evaluation: Assessing the Impact of Public Policy (3 of 4) Objective: Assess policy impact with respect to its costs and benefits • 4.1.3: Nontarget Groups – Definition – Nontarget effects • 4.1.4: Short-Term and Long-Term Effects – Concerns about the effects of a program – Conclusions drawn from impact studies Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 11. 4.1: Policy Evaluation: Assessing the Impact of Public Policy (4 of 4) Objective: Assess policy impact with respect to its costs and benefits • 4.1.5: Calculating Net Benefits and Costs – Formula – Difficulties in coming up with a net balance Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 12. 4.2: The Symbolic Impact of Policy Objective: Identify symbolic impact of a policy as the people's perceptions of government action and attitude – Overview – Systematic policy analysis Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 13. 4.3: Program Evaluation: What Governments Usually Do (1 of 4) Objective: Review some of the ways in which government agencies find out the effectiveness of their programs • 4.3.1: Hearings and Reports – Overview – Disadvantages • 4.3.2: Site Visits – Overview – Issues that are looked into Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 14. 4.3: Program Evaluation: What Governments Usually Do (2 of 4) Objective: Review some of the ways in which government agencies find out the effectiveness of their programs • 4.3.3: Program Measures – Overview – Examples of policy output measures • 4.3.4: Comparison with Professional Standards – Overview – Disadvantages Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 15. 4.3: Program Evaluation: What Governments Usually Do (3 of 4) Objective: Review some of the ways in which government agencies find out the effectiveness of their programs • 4.3.5: Evaluation of Citizens' Complaints – Overview – Disadvantages • 4.3.6: Surveys of Public Opinion – Overview – Polls Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 16. 4.3: Program Evaluation: What Governments Usually Do (4 of 4) Objective: Review some of the ways in which government agencies find out the effectiveness of their programs • 4.3.7: Congressional Investigations – Overview – Process – Role of congressional committees – Effects Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 17. 4.4: Program Evaluation: What Governments Can Do (1 of 4) Objective: Recall the process of systematic program evaluation – Failure of the common evaluative methods – Systematic program evaluation Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 18. 4.4: Program Evaluation: What Governments Can Do (2 of 4) Objective: Recall the process of systematic program evaluation • 4.4.1: Before Versus After Comparisons – Overview – Disadvantages Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 19. Figure 4.1: Research Designs for Policy Evaluation Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 20. 4.4: Program Evaluation: What Governments Can Do (3 of 4) Objective: Recall the process of systematic program evaluation • 4.4.2: Projected Trend Line Versus Postprogram Comparisons – Process of evaluation – Establishing a trend line • 4.4.3: Comparisons Between Jurisdictions With and Without Programs – Overview – Disadvantages – Alternate design Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 21. 4.4: Program Evaluation: What Governments Can Do (4 of 4) Objective: Recall the process of systematic program evaluation • 4.4.4: Comparisons Between Control and Experimental Groups Before and After Program Implementation – Process – Initial requirements – Advantage Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 22. 4.5: Experimental Policy Research (1 of 3) Objective: Analyze arguments against government-sponsored experimental policy research • 4.5.1: A Bias Toward Positive Results – Relevant questions – Consequences of unsupportive or negative results • 4.5.2: The Hawthorne Effect – Overview – Conclusion Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 23. 4.5: Experimental Policy Research (2 of 3) Objective: Analyze arguments against government-sponsored experimental policy research • 4.5.3: Generalizing Results to the Nation – Overview – Guarantee of minimum annual income • 4.5.4: Ethical and Legal Issues – Relevant fields Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 24. 4.5: Experimental Policy Research (3 of 3) Objective: Analyze arguments against government-sponsored experimental policy research • 4.5.5: Political Interpretations of Results – Overview – Relevant fields Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 25. 4.6: Federal Evaluation: The Office of Management and Budget Objective: Outline the responsibilities of the Office of Management and Budget • 4.6.1: Benefit–Cost Analysis of Federal Regulations – Overview – Reasoned determination • 4.6.2: Value of a Statistical Life – Controversies – Relevant agencies Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 26. 4.7: Federal Evaluation: The General Accountability Office Objective: Report the responsibilities of the General Accountability Office – Overview – Criticized agencies Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 27. 4.8: Program Evaluation: Why It Fails So Often Objective: Review some of the problems plaguing policy evaluation – Overview – Requirements Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 28. 4.9: How Bureaucrats Explain Negative Findings Objective: State some of the ways in which bureaucrats try to defend their policy – Positive arguments – General laws Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 29. 4.10: Why Government Programs Are Seldom Terminated (1 of 2) Objective: Analyze reasons for the continuation of inefficient government programs and policies • 4.10.1: Concentrated Benefits, Dispersed Costs – Overview – Characteristics of beneficiaries • 4.10.2: Legislative and Bureaucratic Interests – Incentives for bureaucrats – Actions taken by legislators Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 30. 4.10: Why Government Programs Are Seldom Terminated (2 of 2) Objective: Analyze reasons for the continuation of inefficient government programs and policies • 4.10.3: Incrementalism at Work – Relevant considerations – Role of negative evaluative studies Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 31. 4.11: Politics as a Substitute for Analysis Objective: Identify steps of a political approach to policy analysis – Requirements – Emphasis Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 32. 4.12: The Limits of Public Policy Objective: Evaluate limitations of policies in addressing all problems – Americans’ expectations of governments – Limitations of policy Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
- 33. Summary: Policy Evaluation • Policy evaluation involves assessing impact of policy on target and nontarget groups • Government agencies themselves usually report policy output measures • The OMB and GAO undertake evaluations of federal programs • There are many political and bureaucratic obstacles to effective policy evaluation Copyright © 2017, 2013, 2011 Pearson Education, Inc. All Rights Reserved
Editor's Notes
- Point 1- Overview of policy evaluation Learning about consequences of public policy Stated goals of a program not always clear Notion of policy evaluation thus limited to their achievements Point 2- Policy impact Effects of a policy on real-world conditions Includes its impact on target situations or groups Includes its impact on future and immediate conditions All of its benefits and costs at present and in the future measured
- 4.1.1: Measuring Impact, Not Output Point 1- Definition of policy output Measure of a government activity Important for describing the determinants of a public policy Examples include welfare benefits paid, criminal arrests and prosecutions, Medicare payments, and school enrolments Point 2- Difference between policy impact and policy output Policy output does not tell much about poverty, crime, health, or educational achievement Identifying the changes in society that are associated with measures of government activity necessary in assessing policy impact 4.1.2: Target Groups Point 1- Overview Part of the population for whom a program is intended Should be first identified before determining the desired effects of the program on them Point 2- Various effects of a program on target groups To change knowledge, attitudes, awareness, interests, or behavior To change physical or economic circumstances Other unintended, or side, effects
- 4.1.3: Nontarget Groups Point 1- Definition Various other segments of society influenced by a program or policy unintentionally Identifying important nontarget groups is difficult Point 2- Nontarget effects Could be expressed as beneficial such as the benefits to the construction industry of public housing projects Could also be expressed as costly 4.1.4: Short-Term and Long-Term Effects Point 1- Concerns about the effects of a program Time period before the benefits or costs are felt Whether a program is designed for short-time emergencies or as a long-term development effort Issues that will prevent the processes of incrementalism and bureaucratization from turning it into a long-term program Point 2- Conclusions drawn from impact studies New or innovative programs have short-term positive effects that disappear as the novelty and enthusiasm wear off Other programs experience difficulties at first but turn out to have “sleeper” effects
- 4.1.5: Calculating Net Benefits and Costs Point 1- Formula All of the benefits of a policy, both immediate and long range, minus all the costs, both immediate and future Point 2- Difficulties in coming up with a net balance Knowing all the costs and benefits Everyone agreeing on what is a benefit and what is a cost
- Point 1- Overview Symbolic impact of a policy deals with perceptions individuals have of government action and their attitudes toward it Public policies frequently judged in terms of good intentions Policies help hold people together and maintain an orderly state Point 2- Systematic policy analysis Concentrates on what governments do, why they do it, and what difference it makes Devotes less attention to what governments say
- 4.3.1: Hearings and Reports Point 1- Overview Most common type of program review Government administrators asked by chief executives or legislators to report on the accomplishments of their own programs Program administrators provide written annual reports Point 2- Disadvantages Not considered very objective means of program evaluation Benefits are frequently magnified and costs are subsequently minimized 4.3.2: Site Visits Point 1- Overview Teams of high ranking officers visit agencies or conduct inspections in the field Teams include high-ranking administrators, expert consultants, legislators, or some combination of these people Point 2- Issues that are looked into Management of a program Whether specific guidelines are being followed Availability of competent staffs Whether or not clients are pleased with the services
- 4.3.3: Program Measures Point 1- Overview Output measures are covered in the data developed by the agencies themselves Does not indicate what impact the policies have on society Point 2- Examples of policy output measures Number of recipients in various welfare programs Number of persons in work-force training programs Number of public hospital beds available Tons of garbage collected Number of pupils enrolled 4.3.4: Comparison with Professional Standards Point 1- Overview Comparing actual government outputs with ideal outputs Desired levels of outputs are developed by professional associations Point 2- Disadvantages Does not focus on the impact of government activities on the conditions of target or nontarget groups Standards are developed by professionals who are only guessing at what ideal levels of benefits and services should be
- 4.3.5: Evaluation of Citizens' Complaints Point 1- Overview Analyzing the complaints of citizens Administrators occasionally develop questionnaires for participants in their program to learn whether they are satisfied or not Point 2- Disadvantages Not all citizens voluntarily submit complaints Critics of government programs are self-selected and rarely represent the public Complaints of the vocal few are not always shared by the many more who have not spoken up Does not focus on the impact of the program on the lives of its participants 4.3.6: Surveys of Public Opinion Point 1- Overview Governments survey citizens about their satisfaction or dissatisfaction with various programs and services Common at the local level of government Point 2- Polls Reflect general satisfaction or dissatisfaction with federal programs Usually instructive
- 4.3.7: Congressional Investigations Point 1- Overview An implied constitutional power Derives from Congressional power to legislate Point 2- Process Extensive background research Questioning of witnesses, usually in public Point 3- Role of congressional committees To open investigations to probe particular issues and report on them Make policy reports based on the investigation Can issue subpoenas compelling witnesses to testify Can hold witnesses in contempt of Congress if they refuse to testify Point 4- Effects Mostly publicize an issue or attract publicity to members of Congress Sometimes result in the evaluation of policies and programs
- Point 1- Failure of the common evaluative methods Does not attempt to weigh costs against benefits Difficult to calculate the costs of making specific changes in society More complex and costly methods of program evaluation are required Point 2- Systematic program evaluation Requires measuring what happened and what would have happened without the program Comparing the two conditions of the society
- 4.4.1: Before Versus After Comparisons Point 1- Overview Most common research design in program evaluation Compares results in a jurisdiction at two times—one before program implementation and the other after Point 2- Disadvantages Difficult to know whether the changes observed came about as a result of the program or as a result of other changes occurring at the same time Usually only target groups are examined
- Design 1 – Before vs. After Design 2 – Projected vs. Postprogram Design 3 – With vs. Without Program Design 4 – Control vs. Experimental Groups
- 4.4.2: Projected Trend Line Versus Postprogram Comparisons Point 1- Process of evaluation Project preprogram trends into the postprogram time period Projections can be compared with what actually happened in society after program implementation Difference between the projections based on preprogram trends and actual postprogram data can be attributed to the program Point 2- Establishing a trend line Data on target groups or conditions must be obtained for several time periods before the program was initiated Requires tremendous effort by program evaluators 4.4.3: Comparisons Between Jurisdictions With and Without Programs Point 1- Overview Comparing individuals who have participated in programs with those who have not Comparing cities, states, or nations that have programs with those that do not Point 2- Disadvantages Difficult to attribute differences in participants’ conditions to differences in government programs Sometimes comparisons are made only in the postprogram period Point 3- Alternate design Observing both kinds of jurisdictions before and after program introduction Estimates differences between jurisdictions before program efforts are considered Provides some protection against attributing differences to a particular program
- 4.4.4: Comparisons Between Control and Experimental Groups Before and After Program Implementation Point 1- Process Careful selection of control and experimental groups that are identical in every way Application of the policy to the experimental group only Comparing changes in the experimental group with changes in the control group after the application of the policy Point 2- Initial requirements Control and experimental groups must be identical Preprogram performance of each group must be measured and found to be the same Program must be applied only to the experimental group Point 3- Advantage Estimates changes derived from the effects of other forces in society
- 4.5.1: A Bias Toward Positive Results Point 1- Relevant questions Are government-sponsored research projects predisposed to produce results supportive of popular reform proposals? Are social scientists inclined to produce findings in support of liberal reform measures? Point 2- Consequences of unsupportive or negative results Go back and recode data Redesign research Reevaluate results 4.5.2: The Hawthorne Effect Point 1- Overview Taken from early experiments at the Hawthorne plant of Western Electric Company in Chicago in 1927 Worker output increased with any change in routine Point 2- Conclusion People behave differently when they know they are being watched Example: Students
- 4.5.3: Generalizing Results to the Nation Point 1- Overview Results obtained with small-scale experiments differ from large-scale ones Example: Brief experiment involving small number of families Point 2- Guarantee of minimum annual income Cultural standards might be changed nationwide Might seriously understate expected cost of an economy-wide program 4.5.4: Ethical and Legal Issues Point 1- Relevant fields Government research Medicine Social experimentation
- 4.5.5: Political Interpretations of Results Point 1- Overview Political milieu shapes policy research Politics helps decide what policies and policy alternatives will be studied Point 2- Relevant fields Education Welfare Housing Health
- 4.6.1: Benefit–Cost Analysis of Federal Regulations Point 1- Overview Army Corps of Engineers undertook in 1936 Development of government benefit–cost analysis Point 2- Reasoned determination Benefits justify the costs Regulatory action maximized net societal benefits 4.6.2: Value of a Statistical Life Point 1- Controversies Valuation of a human life Required in the design of evaluations of health and safety regulations Point 2- Relevant agencies Environmental Protection Agency Department of Transportation Food and Drug Administration Occupational Safety and Health Administration Department of Homeland Security
- Point 1- Overview Arm of Congress Audits operations and finances of federal agencies Evaluates programs Reports findings to Congress Point 2- Criticized agencies Defense Department Environmental Protection Agency Social Security
- Point 1- Overview To determine goals of programs To not actually change the conditions of target groups Government agencies usually have a heavy investment Point 2- Requirements Funds Facilities Time Personnel
- Point 1- Positive arguments Effects of the program are long range and cannot be measured at the present time Effects of the program are diffuse and general in nature Effects of the program are subtle and cannot be identified by crude measures or statistics Experimental research cannot be carried out effectively Point 2- General laws Wilson’s First Law Wilson’s Second Law
- 4.10.1: Concentrated Benefits, Dispersed Costs Point 1- Overview Limited benefits concentrated in a small, well-organized constituency Greater costs dispersed over a large, unorganized, uninformed public Point 2- Characteristics of beneficiaries Concerned Well-informed Active in their support 4.10.2: Legislative and Bureaucratic Interests Point 1- Incentives for bureaucrats To resist or undermine negative evaluations of programs To respond to public criticism by making only marginal changes in programs To claim programs are failing because not enough is being spent Point 2- Actions taken by legislators To protect failed programs To minimize reform To block termination
- 4.10.3: Incrementalism at Work Point 1- Relevant considerations Attention focused on proposed changes in existing programs Value of programs in their entirety not considered Point 2- Role of negative evaluative studies Limiting increases for failed programs Identifying programs ripe for budget cutting
- Point 1- Requirements Some agreement on what problems the government should undertake to resolve Some agreement on the nature of societal benefits and costs and the weights to be given Some agreement on the formulation of a research design Point 2- Emphasis Search for common concerns that might form the basis for identification of societal problems Reasonable trade-offs among conflicting values at each stage of the policymaking process Search for mutually beneficial outcomes for diverse groups Bargaining among participants, even in separate policy areas, to win allies Compromise and conciliation and a willingness to accept modest net gains
- Point 1- Americans’ expectations of governments Eliminate poverty End racism Ensure peace Prevent crime Restore cities Point 2- Limitations of policy Some societal problems are incapable of solution Expectations may always outrace the capabilities of governments Policies that solve problems of one group in society may create problems for other groups Possibility that some societal forces cannot be harnessed by governments