This document outlines a training curriculum for evaluating the socio-economic impact of a water program. It covers four sessions over four days: introduction and overview, evaluation design, sample design and data collection, and indicators and questionnaire design. Key topics include causal inference, impact evaluation methods like randomized assignment and difference-in-differences, sample designs, and designing indicators and questionnaires. The document uses a case study of Mexico's Progresa anti-poverty program to illustrate concepts like randomized assignment, pre-post comparisons, and enrolled vs non-enrolled comparisons.
Diseases can have significant economic impacts through costs of disease, public health response costs, direct healthcare costs, and lost productivity. Costs of disease are measured by effects on health and quality of life, healthcare spending, and lost productivity when people are ill or die from infectious diseases. Public health responses to outbreaks like the 2002 West Nile virus epidemic in Louisiana cost $20.1 million. Direct healthcare costs include hospitalization, testing, and treatment, but are often underestimated due to factors like hospital capacity constraints and recovery times. Lost productivity results from deaths, hospitalizations, and outpatient visits, but these losses are also underestimated since they do not account for real-world inefficiencies and delays.
Annual Results and Impact Evaluation Workshop for RBF - Day Six - Introductio...RBFHealth
This document discusses impact evaluation methods for results-based financing (RBF) programs. It aims to build evidence on what works and why through impact evaluations that are built into program operations and involve government ownership. The document outlines key policy questions around whether and how RBF works and discusses methods like randomized assignment that can help estimate the causal impact of RBF programs on outcomes like health service utilization and health impacts. It provides an example impact evaluation of Rwanda's performance-based financing project which found improvements in prenatal care quality, skilled delivery rates, and child preventive care utilization from RBF.
This document discusses causal inference and program evaluation. It notes that evaluating programs requires estimating the counterfactual outcome for participants in the absence of the program, which is difficult. Common problems in evaluation include selection bias if participants differ from non-participants in unobserved ways, spillover effects, and impact heterogeneity. Internal validity assesses if the true impact is measured, while external validity examines generalizability. Estimating average treatment effects requires addressing non-random selection into programs.
1) The document outlines methods for policy evaluation and treatment analysis using non-experimental data, including difference-in-differences (DiD) estimation and propensity score matching.
2) DiD estimation compares the changes in outcomes over time between a treated group and a control group, assuming that in absence of treatment the trends would be parallel. This controls for factors that influence both groups.
3) Propensity score matching matches treated and untreated units based on their propensity scores, which is the probability of being treated estimated from observed covariates. This balances the groups on observed characteristics without losing observations.
1) The document outlines methods for policy evaluation and treatment analysis using non-experimental data, including difference-in-differences (DiD) estimation and propensity score matching.
2) DiD estimation compares the changes in outcomes over time between a treated group and a control group, assuming that in absence of treatment the trends would be parallel. This controls for factors that influence both groups.
3) Propensity score matching matches treated and untreated units based on their propensity scores, which is the probability of being treated estimated from observed covariates. This balances the groups on observed characteristics without losing observations.
HLEG thematic workshop on Measuring Inequalities of Income and Wealth, Nora L...StatsCommunications
HLEG workshop on Measuring Inequalities of Income and Wealth, 15-16 September 2015, Berlin, Germany, More information at: http://oe.cd/hleg-workshop-inequalities-income-and-wealth
International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.
This document outlines a training curriculum for evaluating the socio-economic impact of a water program. It covers four sessions over four days: introduction and overview, evaluation design, sample design and data collection, and indicators and questionnaire design. Key topics include causal inference, impact evaluation methods like randomized assignment and difference-in-differences, sample designs, and designing indicators and questionnaires. The document uses a case study of Mexico's Progresa anti-poverty program to illustrate concepts like randomized assignment, pre-post comparisons, and enrolled vs non-enrolled comparisons.
Diseases can have significant economic impacts through costs of disease, public health response costs, direct healthcare costs, and lost productivity. Costs of disease are measured by effects on health and quality of life, healthcare spending, and lost productivity when people are ill or die from infectious diseases. Public health responses to outbreaks like the 2002 West Nile virus epidemic in Louisiana cost $20.1 million. Direct healthcare costs include hospitalization, testing, and treatment, but are often underestimated due to factors like hospital capacity constraints and recovery times. Lost productivity results from deaths, hospitalizations, and outpatient visits, but these losses are also underestimated since they do not account for real-world inefficiencies and delays.
Annual Results and Impact Evaluation Workshop for RBF - Day Six - Introductio...RBFHealth
This document discusses impact evaluation methods for results-based financing (RBF) programs. It aims to build evidence on what works and why through impact evaluations that are built into program operations and involve government ownership. The document outlines key policy questions around whether and how RBF works and discusses methods like randomized assignment that can help estimate the causal impact of RBF programs on outcomes like health service utilization and health impacts. It provides an example impact evaluation of Rwanda's performance-based financing project which found improvements in prenatal care quality, skilled delivery rates, and child preventive care utilization from RBF.
This document discusses causal inference and program evaluation. It notes that evaluating programs requires estimating the counterfactual outcome for participants in the absence of the program, which is difficult. Common problems in evaluation include selection bias if participants differ from non-participants in unobserved ways, spillover effects, and impact heterogeneity. Internal validity assesses if the true impact is measured, while external validity examines generalizability. Estimating average treatment effects requires addressing non-random selection into programs.
1) The document outlines methods for policy evaluation and treatment analysis using non-experimental data, including difference-in-differences (DiD) estimation and propensity score matching.
2) DiD estimation compares the changes in outcomes over time between a treated group and a control group, assuming that in absence of treatment the trends would be parallel. This controls for factors that influence both groups.
3) Propensity score matching matches treated and untreated units based on their propensity scores, which is the probability of being treated estimated from observed covariates. This balances the groups on observed characteristics without losing observations.
1) The document outlines methods for policy evaluation and treatment analysis using non-experimental data, including difference-in-differences (DiD) estimation and propensity score matching.
2) DiD estimation compares the changes in outcomes over time between a treated group and a control group, assuming that in absence of treatment the trends would be parallel. This controls for factors that influence both groups.
3) Propensity score matching matches treated and untreated units based on their propensity scores, which is the probability of being treated estimated from observed covariates. This balances the groups on observed characteristics without losing observations.
HLEG thematic workshop on Measuring Inequalities of Income and Wealth, Nora L...StatsCommunications
HLEG workshop on Measuring Inequalities of Income and Wealth, 15-16 September 2015, Berlin, Germany, More information at: http://oe.cd/hleg-workshop-inequalities-income-and-wealth
International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.
The document discusses a study on the effects of unconditional cash transfers on labor supply in Zambia. It finds that cash transfers led to a reduction in wage labor but an increase in own farm labor, with the effects being significant at relatively lower/higher transfer levels. Instrumental variable estimates using time to collect transfers and average community transfer size yielded similar results. The study concludes that cash transfers can have incentive effects on labor supply but disincentives are only seen at transfer levels well above actual amounts received.
Ragui Assaad- University of Minnesota
Caroline Krafft- ST. Catherine University
ERF Training on Applied Micro-Econometrics and Public Policy Evaluation
Cairo, Egypt July 25-27, 2016
www.erf.org.eg
Using financial incentives to increase testing uptake versie 2Jennie van de Weerd
The document discusses using financial incentives to increase HIV testing and reduce risk behavior in men in South Africa. It presents an intervention where men are offered $75 to get tested by the Desmund Tutu HIV Foundation. The objectives are to review literature on incentive-based systems and assess the potential cost-effectiveness. A costing model is developed and finds that providing incentives seems cost-effective after 2 years if it reduces the chance of infection below 15%. Intense follow up is still needed to sustain behavior change.
Randomized social experiments evaluate the impact of social policies by comparing outcomes between a treatment group that receives a policy and a control group that does not. Influential experiments have evaluated policies like class size reduction, welfare programs, housing vouchers, and conditional cash transfers. Randomized experiments aim to build counterfactuals by randomly assigning individuals to treatment and control groups, so that any difference in outcomes can be attributed to the policy being evaluated. The document describes two French experiments that used randomization to evaluate the impact of intensive job counseling programs for unemployed individuals, finding counseling increased exit rates to employment but led to displacement effects with no overall benefit.
This document provides an overview of health economics and its role in public health. It begins by defining health economics as using an economic framework to help maximize population health given constrained resources. It then discusses the various analyses health economists perform, including economic evaluations like cost-effectiveness analysis. It provides examples of how economics can inform decisions around public health programs and resource allocation in India. Key points made include that health resources are limited so choices must be made, and that economic evaluations can help identify which health interventions provide the best value. The conclusion emphasizes that health economics should be integrated into health policy and management to help make resource decisions more explicit and fair.
Czarnitzki - Towards a portfolio of additionaliyu indicatorsinnovationoecd
This document discusses current practices in evaluating science, technology, and innovation (STI) policies using econometric methods and identifies areas for future improvement. It notes that methods like matching, difference-in-differences, and instrumental variables are now commonly used to establish control groups and estimate policy impacts. However, greater emphasis is needed on exploring heterogeneous treatment effects across different policy instruments, firm characteristics, and over time to better inform policy design. Indirect effects also need consideration. Improving identification strategies through techniques like regression discontinuity and exploiting natural experiments can further strengthen evaluations.
This document provides an introduction to quantitative impact evaluation methods. It discusses why impact evaluations are important, how to design an evaluation, and common evaluation tools and methodologies. Key points include: impact evaluations measure a program's causal effects, require a comparison group to estimate counterfactual outcomes, and use methods like randomization, matching, regression discontinuity, and difference-in-differences to construct valid comparisons. The goals of evaluations are to measure impacts, assess cost-effectiveness, and explain which program components are most effective.
Pension (In)Stability of Social Security ReformGRAPE
In this paper we consider an economy populated by overlapping generations, who vote on abolishing the funded system and replacing it with the pay-as-you-go scheme (i.e. unprivatizing the pension system). We compare politically stable and politically unstable reforms and show that even if the funded system is overall welfare enhancing, the cohort distribution of benefits along the transition path turns unprivatizing social security politically favorable.
Political (In)Stability of Social Security ReformGRAPE
We analyze the political stability of welfare enhancing privatization of the social security. We consider an economy populated by overlapping generations, who vote on abolishing the funded system and replacing it with the pay-as-you-go scheme, i.e. “unprivatizing” the pension system. We show that even if abolishing the system reduces overall welfare, the distribution of benefits across cohorts along the transition path implies that some ways of “unprivatizing” social security are always politically favored
Does transfer to intensive care units reduce mortality for deteriorating ward...cheweb1
1) The study uses instrumental variable methods to examine the effect of transfer to intensive care units (ICU) on mortality for deteriorating ward patients using data from the UK.
2) The instrumental variable is ICU bed availability, which is hypothesized to directly affect the probability of ICU transfer but be independent of patient mortality.
3) Propensity score matching is also used to increase the strength of the instrumental variable by matching patients exposed to many versus few available ICU beds who are similar on observed characteristics.
Traditional randomized experiments allow us to determine the overall causal impact of a treatment program (e.g. marketing, medical, social, education, political). Uplift modeling (also known as true lift, net lift, incremental lift) takes a further step to identify individuals who are truly positively influenced by a treatment through data mining / machine learning. This technique allows us to identify the “persuadables” and thus optimize target selection in order to maximize treatment benefits. This important subfield of data mining/data science/business analytics has gained significant attention in areas such as personalized marketing, personalized medicine, and political election with plenty of publications and presentations appeared in recent years from both industry practitioners and academics.
In this workshop, I will introduce the concept of Uplift, review existing methods, contrast with the traditional approach, and introduce a new method that can be implemented with standard software. A method and metrics for model assessment will be recommended. Our discussion will include new approaches to handling a general situation where only observational data are available, i.e. without randomized experiments, using techniques from causal inference. Additionally, an integrated modeling approach for uplift and direct response (where it can be identified who actually responded, e.g., click-through or coupon scanning) will be discussed. Last but not least, extension to the multiple treatment situation with solutions to optimizing treatments at the individual level will also be discussed. While the talk is geared towards marketing applications (“personalized marketing”), the same methodologies can be readily applied in other fields such as insurance, medicine, education, political, and social programs. Examples from the retail and non-profit industries will be used to illustrate the methodologies.
DirectionsSet up your IBM SPSS account and run several statisti.docxjakeomoore75037
Directions:
Set up your IBM SPSS account and run several statistical outputs based on the "SPSS Database" Use "Setting Up My SPSS" to set up your SPSS program on your computer or device. You may also use programs such as Laerd Statistics or Intellectus, if you subscribe to them.
The patient outcome or dependent variables and the level of measurement must be displayed in a comparison table which you will provide as an Appendix to the paper. Refer to the "Comparison Table of the Variable's Level of Measurement."
Submit a 1,000-1,250 word data analysis paper outlining the procedures used to analyze the parametric and non-parametric variables in the mock data, the statistics reported, and a conclusion of the results.
Provide a conclusive result of the data analyses based on the guidelines below for statistical significance.
PAIRED SAMPLE T-TEST: Identify the variables BaselineWeight and InterventionWeight. Using the Analysis menu in SPSS, go to Compare Means, Go to the Paired Sample t-test. Add the BaselineWeight and InterventionWeight in the Pair 1 fields. Click OK. Report the mean weights, standard deviations, t-statistic, degrees of freedom, and p level. Report as t(df)=value, p = value. Report the p level out three digits.
INDEPENDENT SAMPLE T-TEST: Identify the variables InterventionGroups and PatientWeight. Go to the Analysis Menu, go to Compare Means, Go to Independent Samples tT-test. Add InterventionGroups to the Grouping Factor. Define the groups according to codings in the variable view (1=Intervention, 2 =Baseline). Add PatientWeight to the test variable field. Click OK. Report the mean weights, standard deviations, t-statistic, degrees of freedom, and p level. Report t(df)=value, p = value. Report the p level out three digits
CHI-SQUARE (Independent): Identify the variables BaselineReadmission and InterventionReadmission. Go to the Analysis Menu, go to Descriptive Statistics, go to Crosstabs. Add BaselineReadmission to the row and InterventionReadmission to the column. Click the Statistics button and choose Chi-Square. Select eta to report the Effect Size. Click suppress tables. Click OK. Report the frequencies of the total events, the chi-square statistic, degrees of freedom, and p level. Report ꭓ2 (df) =value, p =value. Report the p level out three digits.
MCNEMAR (Paired): Identify the variables BaselineCompliance and InterventionCompliance. Go to the Analysis Menu, go to Descriptive Statistics, go to Crosstabs. Add BaselineCompliance to the row and InterventionCompliance to the column. Click the Statistics button and choose Chi-Square and McNemars. Select eta to report the Effect Size. Click suppress tables. Click OK. Report the frequencies of the events, the Chi-square, and the McNemar’s p level. Report (p =value). Report the p level out three digits.
MANN WHITNEY U: Identify the variables InterventionGroups and PatientSatisfaction. Using the Analysis Menu, go to Non-parametric Statistics, go to LegacyDialogs, go to 2 I.
Impact evaluation in-depth: More on methods and example of impact evaluation ...CIFOR-ICRAF
Presented by Colas Chervier (CIRAD) at "Workshop on impact evaluation methods and research collaboration kick-off", Samarinda, Indonesia, on 10 October 2022
1. Economic evaluation of substance abuse treatment programs aims to assess costs and benefits in order to improve effectiveness and rationalize resource allocation.
2. Key aspects of economic evaluation include measuring treatment costs using tools like DATCAP and measuring outcomes using tools like ASI to assess areas like criminal behavior, employment, and health service use.
3. Benefits are converted to monetary values and compared to costs to calculate cost-benefit ratios and net benefits to understand the monetary gains from investments in reducing crime, improving health, and reducing costs to the justice system.
P A P A N A S T A S A T O S Economic EvaluationMark de Haan
1. Economic evaluation of substance abuse treatment programs aims to assess costs and benefits in order to improve effectiveness and rationalize resource allocation.
2. Key aspects of economic evaluation include measuring treatment costs using tools like DATCAP and measuring outcomes using tools like ASI to assess areas like criminal behavior, employment, and health service use.
3. Benefits are converted to monetary values and compared to costs to calculate cost-benefit ratios and net benefits to understand the monetary gains from investments in reducing crime, improving health, and reducing costs to the justice system.
The document provides an overview of a workshop on applying system dynamics methods to understand complex adaptive systems in health. The workshop objectives are to introduce the complex adaptive systems framework, provide hands-on experience with system dynamics software, and discuss how system dynamics can be applied to research. The workshop outline includes introductions to complex adaptive systems and system dynamics, having participants build their own models, and a discussion session.
The document discusses evaluating the impact of interventions using a counterfactual approach. It introduces key concepts like the potential outcomes model, selection bias, and strategies to reduce bias like randomization, differences-in-differences, and propensity score matching. As a case study, it evaluates the impact of Italy's Credit Guarantee Fund for SMEs, which provides partial guarantees on bank loans. The evaluation uses loan application data and financial statements for firms to compare loan amounts for approved versus rejected firms using instrumental variables, differences-in-differences, and propensity score matching to estimate the additional borrowing attributable to the guarantees.
The document summarizes a study that assessed the impacts of a random-audits program in Brazil that aimed to reduce corruption within health transfers. The study found that the program significantly reduced procurement problems but health outputs and outcomes worsened, as mismanagement increased proportionally to the decrease in corruption. Transfers that were more procurement-intensive saw larger decreases in reported corruption after the program, but no changes in potential outcomes, suggesting the program was effective at reducing corruption specifically.
This document provides an overview of cost-benefit analysis as an economic evaluation technique. It discusses key aspects of CBA including:
- Measuring both costs and benefits in monetary terms, with an intervention undertaken if benefits exceed costs.
- Perspectives taken including patient, provider, and societal.
- Methods for measuring costs such as direct, indirect, and intangible costs.
- Methods for measuring monetary value of benefits including cost of illness averted and contingent valuation using willingness to pay.
- An example cost-benefit analysis of interventions to reduce cholera cases in an area.
The document outlines Leo Guelman's PhD thesis on developing improved statistical and algorithmic methods for personalized treatment learning (PTL) models, with applications to insurance. It introduces PTL, which aims to select the optimal treatment for each individual based on their characteristics. The thesis contributes novel PTL methods like uplift random forests and causal conditional inference trees, and shows they outperform existing approaches in simulations. It also applies PTL to optimize insurance marketing strategies using experimental data from a large insurer.
Setting a Path for Improved Health Outcomes RBFRBFHealth
Learning is a critical part of the HRITF RBF portfolio, with all programs benefiting from an embedded impact evaluation and in some cases, complemented by qualitative research components such as process evaluation studies. The presentation discusses the following topics:
1. Using RBF at the community-level to address demand side barriers
This presentation elaborates on the early evidence and the rationale for using RBF at the community level. It will share lessons learned from the implementation of community RBF at country level.
2. Using RBF to Strengthen Quality of Care: Early Lessons
This presentation discusses the broader policy implications of using RBF to strengthen the quality of care. It will explore how Measuring and Paying for the Quality of Care has been operationalized and will highlight the experience of Nigeria. Lastly, it will focus on measuring and Analyzing the Quality of Care from the Impact Evaluation perspective.
Cost-Effectiveness Analysis of RBF in Zimbabwe and ZambiaRBFHealth
Profs. Shepard and Zeng have been leading projects for the Bank to develop methods for performing a cost-effectiveness analysis of Results-Based Financing (RBF) programs and applying them to maternal-child health (MCH) services in Zambia and Zimbabwe. Both countries’ RBF programs proved highly cost-effective. Methods and results should be informative to other RBF and MCH programs.
More Related Content
Similar to Annual Results and Impact Evaluation Workshop for RBF - Day Two - Measuring Impact - Impact Evaluation Methods for Policy Makers - A Refresher
The document discusses a study on the effects of unconditional cash transfers on labor supply in Zambia. It finds that cash transfers led to a reduction in wage labor but an increase in own farm labor, with the effects being significant at relatively lower/higher transfer levels. Instrumental variable estimates using time to collect transfers and average community transfer size yielded similar results. The study concludes that cash transfers can have incentive effects on labor supply but disincentives are only seen at transfer levels well above actual amounts received.
Ragui Assaad- University of Minnesota
Caroline Krafft- ST. Catherine University
ERF Training on Applied Micro-Econometrics and Public Policy Evaluation
Cairo, Egypt July 25-27, 2016
www.erf.org.eg
Using financial incentives to increase testing uptake versie 2Jennie van de Weerd
The document discusses using financial incentives to increase HIV testing and reduce risk behavior in men in South Africa. It presents an intervention where men are offered $75 to get tested by the Desmund Tutu HIV Foundation. The objectives are to review literature on incentive-based systems and assess the potential cost-effectiveness. A costing model is developed and finds that providing incentives seems cost-effective after 2 years if it reduces the chance of infection below 15%. Intense follow up is still needed to sustain behavior change.
Randomized social experiments evaluate the impact of social policies by comparing outcomes between a treatment group that receives a policy and a control group that does not. Influential experiments have evaluated policies like class size reduction, welfare programs, housing vouchers, and conditional cash transfers. Randomized experiments aim to build counterfactuals by randomly assigning individuals to treatment and control groups, so that any difference in outcomes can be attributed to the policy being evaluated. The document describes two French experiments that used randomization to evaluate the impact of intensive job counseling programs for unemployed individuals, finding counseling increased exit rates to employment but led to displacement effects with no overall benefit.
This document provides an overview of health economics and its role in public health. It begins by defining health economics as using an economic framework to help maximize population health given constrained resources. It then discusses the various analyses health economists perform, including economic evaluations like cost-effectiveness analysis. It provides examples of how economics can inform decisions around public health programs and resource allocation in India. Key points made include that health resources are limited so choices must be made, and that economic evaluations can help identify which health interventions provide the best value. The conclusion emphasizes that health economics should be integrated into health policy and management to help make resource decisions more explicit and fair.
Czarnitzki - Towards a portfolio of additionaliyu indicatorsinnovationoecd
This document discusses current practices in evaluating science, technology, and innovation (STI) policies using econometric methods and identifies areas for future improvement. It notes that methods like matching, difference-in-differences, and instrumental variables are now commonly used to establish control groups and estimate policy impacts. However, greater emphasis is needed on exploring heterogeneous treatment effects across different policy instruments, firm characteristics, and over time to better inform policy design. Indirect effects also need consideration. Improving identification strategies through techniques like regression discontinuity and exploiting natural experiments can further strengthen evaluations.
This document provides an introduction to quantitative impact evaluation methods. It discusses why impact evaluations are important, how to design an evaluation, and common evaluation tools and methodologies. Key points include: impact evaluations measure a program's causal effects, require a comparison group to estimate counterfactual outcomes, and use methods like randomization, matching, regression discontinuity, and difference-in-differences to construct valid comparisons. The goals of evaluations are to measure impacts, assess cost-effectiveness, and explain which program components are most effective.
Pension (In)Stability of Social Security ReformGRAPE
In this paper we consider an economy populated by overlapping generations, who vote on abolishing the funded system and replacing it with the pay-as-you-go scheme (i.e. unprivatizing the pension system). We compare politically stable and politically unstable reforms and show that even if the funded system is overall welfare enhancing, the cohort distribution of benefits along the transition path turns unprivatizing social security politically favorable.
Political (In)Stability of Social Security ReformGRAPE
We analyze the political stability of welfare enhancing privatization of the social security. We consider an economy populated by overlapping generations, who vote on abolishing the funded system and replacing it with the pay-as-you-go scheme, i.e. “unprivatizing” the pension system. We show that even if abolishing the system reduces overall welfare, the distribution of benefits across cohorts along the transition path implies that some ways of “unprivatizing” social security are always politically favored
Does transfer to intensive care units reduce mortality for deteriorating ward...cheweb1
1) The study uses instrumental variable methods to examine the effect of transfer to intensive care units (ICU) on mortality for deteriorating ward patients using data from the UK.
2) The instrumental variable is ICU bed availability, which is hypothesized to directly affect the probability of ICU transfer but be independent of patient mortality.
3) Propensity score matching is also used to increase the strength of the instrumental variable by matching patients exposed to many versus few available ICU beds who are similar on observed characteristics.
Traditional randomized experiments allow us to determine the overall causal impact of a treatment program (e.g. marketing, medical, social, education, political). Uplift modeling (also known as true lift, net lift, incremental lift) takes a further step to identify individuals who are truly positively influenced by a treatment through data mining / machine learning. This technique allows us to identify the “persuadables” and thus optimize target selection in order to maximize treatment benefits. This important subfield of data mining/data science/business analytics has gained significant attention in areas such as personalized marketing, personalized medicine, and political election with plenty of publications and presentations appeared in recent years from both industry practitioners and academics.
In this workshop, I will introduce the concept of Uplift, review existing methods, contrast with the traditional approach, and introduce a new method that can be implemented with standard software. A method and metrics for model assessment will be recommended. Our discussion will include new approaches to handling a general situation where only observational data are available, i.e. without randomized experiments, using techniques from causal inference. Additionally, an integrated modeling approach for uplift and direct response (where it can be identified who actually responded, e.g., click-through or coupon scanning) will be discussed. Last but not least, extension to the multiple treatment situation with solutions to optimizing treatments at the individual level will also be discussed. While the talk is geared towards marketing applications (“personalized marketing”), the same methodologies can be readily applied in other fields such as insurance, medicine, education, political, and social programs. Examples from the retail and non-profit industries will be used to illustrate the methodologies.
DirectionsSet up your IBM SPSS account and run several statisti.docxjakeomoore75037
Directions:
Set up your IBM SPSS account and run several statistical outputs based on the "SPSS Database" Use "Setting Up My SPSS" to set up your SPSS program on your computer or device. You may also use programs such as Laerd Statistics or Intellectus, if you subscribe to them.
The patient outcome or dependent variables and the level of measurement must be displayed in a comparison table which you will provide as an Appendix to the paper. Refer to the "Comparison Table of the Variable's Level of Measurement."
Submit a 1,000-1,250 word data analysis paper outlining the procedures used to analyze the parametric and non-parametric variables in the mock data, the statistics reported, and a conclusion of the results.
Provide a conclusive result of the data analyses based on the guidelines below for statistical significance.
PAIRED SAMPLE T-TEST: Identify the variables BaselineWeight and InterventionWeight. Using the Analysis menu in SPSS, go to Compare Means, Go to the Paired Sample t-test. Add the BaselineWeight and InterventionWeight in the Pair 1 fields. Click OK. Report the mean weights, standard deviations, t-statistic, degrees of freedom, and p level. Report as t(df)=value, p = value. Report the p level out three digits.
INDEPENDENT SAMPLE T-TEST: Identify the variables InterventionGroups and PatientWeight. Go to the Analysis Menu, go to Compare Means, Go to Independent Samples tT-test. Add InterventionGroups to the Grouping Factor. Define the groups according to codings in the variable view (1=Intervention, 2 =Baseline). Add PatientWeight to the test variable field. Click OK. Report the mean weights, standard deviations, t-statistic, degrees of freedom, and p level. Report t(df)=value, p = value. Report the p level out three digits
CHI-SQUARE (Independent): Identify the variables BaselineReadmission and InterventionReadmission. Go to the Analysis Menu, go to Descriptive Statistics, go to Crosstabs. Add BaselineReadmission to the row and InterventionReadmission to the column. Click the Statistics button and choose Chi-Square. Select eta to report the Effect Size. Click suppress tables. Click OK. Report the frequencies of the total events, the chi-square statistic, degrees of freedom, and p level. Report ꭓ2 (df) =value, p =value. Report the p level out three digits.
MCNEMAR (Paired): Identify the variables BaselineCompliance and InterventionCompliance. Go to the Analysis Menu, go to Descriptive Statistics, go to Crosstabs. Add BaselineCompliance to the row and InterventionCompliance to the column. Click the Statistics button and choose Chi-Square and McNemars. Select eta to report the Effect Size. Click suppress tables. Click OK. Report the frequencies of the events, the Chi-square, and the McNemar’s p level. Report (p =value). Report the p level out three digits.
MANN WHITNEY U: Identify the variables InterventionGroups and PatientSatisfaction. Using the Analysis Menu, go to Non-parametric Statistics, go to LegacyDialogs, go to 2 I.
Impact evaluation in-depth: More on methods and example of impact evaluation ...CIFOR-ICRAF
Presented by Colas Chervier (CIRAD) at "Workshop on impact evaluation methods and research collaboration kick-off", Samarinda, Indonesia, on 10 October 2022
1. Economic evaluation of substance abuse treatment programs aims to assess costs and benefits in order to improve effectiveness and rationalize resource allocation.
2. Key aspects of economic evaluation include measuring treatment costs using tools like DATCAP and measuring outcomes using tools like ASI to assess areas like criminal behavior, employment, and health service use.
3. Benefits are converted to monetary values and compared to costs to calculate cost-benefit ratios and net benefits to understand the monetary gains from investments in reducing crime, improving health, and reducing costs to the justice system.
P A P A N A S T A S A T O S Economic EvaluationMark de Haan
1. Economic evaluation of substance abuse treatment programs aims to assess costs and benefits in order to improve effectiveness and rationalize resource allocation.
2. Key aspects of economic evaluation include measuring treatment costs using tools like DATCAP and measuring outcomes using tools like ASI to assess areas like criminal behavior, employment, and health service use.
3. Benefits are converted to monetary values and compared to costs to calculate cost-benefit ratios and net benefits to understand the monetary gains from investments in reducing crime, improving health, and reducing costs to the justice system.
The document provides an overview of a workshop on applying system dynamics methods to understand complex adaptive systems in health. The workshop objectives are to introduce the complex adaptive systems framework, provide hands-on experience with system dynamics software, and discuss how system dynamics can be applied to research. The workshop outline includes introductions to complex adaptive systems and system dynamics, having participants build their own models, and a discussion session.
The document discusses evaluating the impact of interventions using a counterfactual approach. It introduces key concepts like the potential outcomes model, selection bias, and strategies to reduce bias like randomization, differences-in-differences, and propensity score matching. As a case study, it evaluates the impact of Italy's Credit Guarantee Fund for SMEs, which provides partial guarantees on bank loans. The evaluation uses loan application data and financial statements for firms to compare loan amounts for approved versus rejected firms using instrumental variables, differences-in-differences, and propensity score matching to estimate the additional borrowing attributable to the guarantees.
The document summarizes a study that assessed the impacts of a random-audits program in Brazil that aimed to reduce corruption within health transfers. The study found that the program significantly reduced procurement problems but health outputs and outcomes worsened, as mismanagement increased proportionally to the decrease in corruption. Transfers that were more procurement-intensive saw larger decreases in reported corruption after the program, but no changes in potential outcomes, suggesting the program was effective at reducing corruption specifically.
This document provides an overview of cost-benefit analysis as an economic evaluation technique. It discusses key aspects of CBA including:
- Measuring both costs and benefits in monetary terms, with an intervention undertaken if benefits exceed costs.
- Perspectives taken including patient, provider, and societal.
- Methods for measuring costs such as direct, indirect, and intangible costs.
- Methods for measuring monetary value of benefits including cost of illness averted and contingent valuation using willingness to pay.
- An example cost-benefit analysis of interventions to reduce cholera cases in an area.
The document outlines Leo Guelman's PhD thesis on developing improved statistical and algorithmic methods for personalized treatment learning (PTL) models, with applications to insurance. It introduces PTL, which aims to select the optimal treatment for each individual based on their characteristics. The thesis contributes novel PTL methods like uplift random forests and causal conditional inference trees, and shows they outperform existing approaches in simulations. It also applies PTL to optimize insurance marketing strategies using experimental data from a large insurer.
Similar to Annual Results and Impact Evaluation Workshop for RBF - Day Two - Measuring Impact - Impact Evaluation Methods for Policy Makers - A Refresher (20)
Setting a Path for Improved Health Outcomes RBFRBFHealth
Learning is a critical part of the HRITF RBF portfolio, with all programs benefiting from an embedded impact evaluation and in some cases, complemented by qualitative research components such as process evaluation studies. The presentation discusses the following topics:
1. Using RBF at the community-level to address demand side barriers
This presentation elaborates on the early evidence and the rationale for using RBF at the community level. It will share lessons learned from the implementation of community RBF at country level.
2. Using RBF to Strengthen Quality of Care: Early Lessons
This presentation discusses the broader policy implications of using RBF to strengthen the quality of care. It will explore how Measuring and Paying for the Quality of Care has been operationalized and will highlight the experience of Nigeria. Lastly, it will focus on measuring and Analyzing the Quality of Care from the Impact Evaluation perspective.
Cost-Effectiveness Analysis of RBF in Zimbabwe and ZambiaRBFHealth
Profs. Shepard and Zeng have been leading projects for the Bank to develop methods for performing a cost-effectiveness analysis of Results-Based Financing (RBF) programs and applying them to maternal-child health (MCH) services in Zambia and Zimbabwe. Both countries’ RBF programs proved highly cost-effective. Methods and results should be informative to other RBF and MCH programs.
Evaluation of the Rwanda Community Performance-Based Financing ProgramRBFHealth
This study evaluates the impact of two interventions introduced as part of the Rwanda Community Performance-Based Financing Program to increase coverage of targeted maternal and child health services: rewards to cooperatives of community health workers and demand-side conditional in-kind transfers. The evaluation exploits experimental design with intervention randomly assigned at the sub-district level for a duration of two and a half years. The analysis finds no impact of the incentives to cooperatives of community health workers. However, conditional in-kind demand-side incentives are shown to significantly increase take up of timely antenatal and postnatal consultations.
RBF districts saw increases in some key health indicators compared to control districts, including institutional deliveries and the timing of first antenatal care visits. Quality of services improved in some areas for RBF districts such as structural quality indices and availability of some delivery equipment. Health systems were strengthened under RBF, with increases in facility governance, autonomy, and job satisfaction reported compared to control districts. The results provide evidence that RBF had a positive impact on priority health services in Zambia.
Performance-based financing of maternal and child health: non-experimental ev...RBFHealth
Ellen Van de Poel presents the findings of two studies that evaluate the impact of Performance-Based Financing (PBF) in Burundi and Cambodia. Both studies exploit the geographic expansion of PBF to estimate its effect on the utilization of maternal and child health services using data from the Demographic Health Surveys.
Success beyond numbers: The Salud Mesoamerica Initiative’s results-based fin...RBFHealth
The Salud Mesoamérica Initiative (SMI) is a public-private partnership between the Bill & Melinda Gates Foundation (BMGF), the Carlos Slim Foundation, the Government of Spain, the Inter-American Development Bank (IDB), the countries of Central America and the State of Chiapas in Mexico. SMI works to reduce maternal and health inequalities through a results-based financing model, aligned with priorities established by the governments of the region. Among the poor in Mesoamerica, only 5 out of every 10 pregnant women are attended during childbirth by skilled birth personnel and the mortality rate among children in poverty is twice that of the national average.
Building agile and data-driven health system is a fundamental component of the Universal Health Coverage agenda. Data-driven health systems will require that countries set up smart data systems coupled with dynamic and reactive health financing instruments.
A large number of the low-income countries enhance their strategic purchasing function with the introduction of Results-Based Financing (RBF) instruments. In this presentation, Nicolas de Borman describes how the combination of DHIS2 platforms, RBF and mobile devices help build strategic purchasing function in health systems.
Sumar Program's Universal Coverage: Achievements & New Goals Towards 2020RBFHealth
A presentation by Martín Sabignoso of Argentina's Ministry of Health delivered at the RBF Health Seminar, QOn the Road to Effective Universal Health Coverage: What’s New in Argentina’s Use of Performance Incentives? on June 11, 2015.
Long run effects of temporary incentives on medical care productivity in Arge...RBFHealth
A presentation by Pablo Celhay, Paul Gertler, Paula Giovagnoli and Christel Vermeersch, delivered at the RBF Health Seminar, On the Road to Effective Universal Health Coverage: What’s New in Argentina’s Use of Performance Incentives? on June 11, 2015.
Qualitative Research in Results-Based Financing: The Promise and The RealityRBFHealth
This document summarizes a review of qualitative research conducted as part of results-based financing (RBF) studies. The review assessed 17 RBF studies and found that while qualitative research holds promise to understand complex health systems, several challenges exist in practice. Studies often lacked depth in contextualization, sampling, and data collection methods. Opportunities exist to strengthen qualitative RBF research through more formative work to inform study design, locally grounded conceptual frameworks, strengthened fieldwork practices, and smaller, more focused qualitative components.
Effect of Voucher Programs on Utilization, Out-of-Pocket Expenditure and Qual...RBFHealth
A presentation by Timothy Abuya, delivered at the RBF Health Seminar, The Role of Vouchers in Serving Disadvantaged Populations and Improving Quality of Care.
A presentation by Ben Bellows, delivered at the RBF Health Seminar, The Role of Vouchers in Serving Disadvantaged Populations and Improving Quality of Care.
WHO Implementation Research Program on Factors Explaining Success and Failure...RBFHealth
This document discusses implementation research on scaling up Results-Based Financing (RBF) programs from pilot schemes to integrated national health systems. It calls for case studies on RBF initiatives in select low and middle-income countries to identify factors that enable or hinder this transition. Selected proposals will examine RBF scale-up experiences in multiple countries. Next steps include a protocol development workshop to design the research and analyze findings to draw cross-cutting lessons on scaling up RBF.
The Science of Delivery: Use of Administrative Data in The HRITF PortfolioRBFHealth
A presentation by Ha Thi Hong Nguyen, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014. This event was hosted by the Health Results Innovation Trust Fund at The World Bank, in partnership with the PBF Community of Practice in Africa.
PBF Conceptual Framework and Illustration with The Case of NigeriaRBFHealth
A presentation by Dinesh Nair, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014. This event was hosted by the Health Results Innovation Trust Fund at The World Bank, in partnership with the PBF Community of Practice in Africa.
Zimbabwe: Results-Based Financing Improves Coverage, Quality and Financial Pr...RBFHealth
A presentation by Dr. Gwinji, Permanent Secretary, Ministry of Health, Zimbabwe and Dr. Tafadzwa Goverwa- Sibanda, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014. This event was hosted by the Health Results Innovation Trust Fund at The World Bank, in partnership with the PBF Community of Practice in Africa.
India - Karnataka: An Experimental Evaluation of Government Health Insurance ...RBFHealth
A presentation by Somil Nagpal, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014. This event was hosted by the Health Results Innovation Trust Fund at The World Bank, in partnership with the PBF Community of Practice in Africa.
What Are Results-Based Financing Programs Doing Around The World, State of Th...RBFHealth
A presentation by Dinesh Nair, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014. This event was hosted by the Health Results Innovation Trust Fund at The World Bank, in partnership with the PBF Community of Practice in Africa.
A presentation by Bruno Meessen, delivered during "Transforming Health Systems Through Results-Based Financing," an event held during the Third Global Symposium on Health Systems Research in Cape Town on September 30, 2014.
Providing Health in Difficult Contexts: Pre-Pilot Performance-Based Financing...RBFHealth
The Adamawa Primary Health Care System in Nigeria has implemented performance-based financing (PBF) to address underlying issues plaguing the health system. After two years of pre-pilot implementation, results have been encouraging with improvements in key indicators like institutional deliveries and vaccination rates. Success stories like Mayo-Ine health center demonstrate how community engagement and strengthened management can boost coverage. However, some indicators still show room for growth, and deeper analysis finds issues like staffing shortages and infrastructure problems influencing performance. Moving forward, continued scale-up and addressing broader health system challenges will be important to sustain gains under PBF in Adamawa State.
LGBTQ+ Adults: Unique Opportunities and Inclusive Approaches to CareVITASAuthor
This webinar helps clinicians understand the unique healthcare needs of the LGBTQ+ community, primarily in relation to end-of-life care. Topics include social and cultural background and challenges, healthcare disparities, advanced care planning, and strategies for reaching the community and improving quality of care.
Deep Leg Vein Thrombosis (DVT): Meaning, Causes, Symptoms, Treatment, and Mor...The Lifesciences Magazine
Deep Leg Vein Thrombosis occurs when a blood clot forms in one or more of the deep veins in the legs. These clots can impede blood flow, leading to severe complications.
Stem Cell Solutions: Dr. David Greene's Path to Non-Surgical Cardiac CareDr. David Greene Arizona
Explore the groundbreaking work of Dr. David Greene, a pioneer in regenerative medicine, who is revolutionizing the field of cardiology through stem cell therapy in Arizona. This ppt delves into how Dr. Greene's innovative approach is providing non-surgical, effective treatments for heart disease, using the body's own cells to repair heart damage and improve patient outcomes. Learn about the science behind stem cell therapy, its benefits over traditional cardiac surgeries, and the promising future it holds for modern medicine. Join us as we uncover how Dr. Greene's commitment to stem cell research and therapy is setting new standards in healthcare and offering new hope to cardiac patients.
TEST BANK For Accounting Information Systems, 3rd Edition by Vernon Richardso...rightmanforbloodline
TEST BANK For Accounting Information Systems, 3rd Edition by Vernon Richardson, Verified Chapters 1 - 18, Complete Newest Version
TEST BANK For Accounting Information Systems, 3rd Edition by Vernon Richardson, Verified Chapters 1 - 18, Complete Newest Version
TEST BANK For Accounting Information Systems, 3rd Edition by Vernon Richardson, Verified Chapters 1 - 18, Complete Newest Version
This particular slides consist of- what is hypotension,what are it's causes and it's effect on body, risk factors, symptoms,complications, diagnosis and role of physiotherapy in it.
This slide is very helpful for physiotherapy students and also for other medical and healthcare students.
Here is the summary of hypotension:
Hypotension, or low blood pressure, is when the pressure of blood circulating in the body is lower than normal or expected. It's only a problem if it negatively impacts the body and causes symptoms. Normal blood pressure is usually between 90/60 mmHg and 120/80 mmHg, but pressures below 90/60 are generally considered hypotensive.
Joker Wigs has been a one-stop-shop for hair products for over 26 years. We provide high-quality hair wigs, hair extensions, hair toppers, hair patch, and more for both men and women.
Letter to MREC - application to conduct studyAzreen Aj
Application to conduct study on research title 'Awareness and knowledge of oral cancer and precancer among dental outpatient in Klinik Pergigian Merlimau, Melaka'
Unlocking the Secrets to Safe Patient Handling.pdfLift Ability
Furthermore, the time constraints and workload in healthcare settings can make it challenging for caregivers to prioritise safe patient handling Australia practices, leading to shortcuts and increased risks.
Gemma Wean- Nutritional solution for Artemiasmuskaan0008
GEMMA Wean is a high end larval co-feeding and weaning diet aimed at Artemia optimisation and is fortified with a high level of proteins and phospholipids. GEMMA Wean provides the early weaned juveniles with dedicated fish nutrition and is an ideal follow on from GEMMA Micro or Artemia.
GEMMA Wean has an optimised nutritional balance and physical quality so that it flows more freely and spreads readily on the water surface. The balance of phospholipid classes to- gether with the production technology based on a low temperature extrusion process improve the physical aspect of the pellets while still retaining the high phospholipid content.
GEMMA Wean is available in 0.1mm, 0.2mm and 0.3mm. There is also a 0.5mm micro-pellet, GEMMA Wean Diamond, which covers the early nursery stage from post-weaning to pre-growing.
Empowering ACOs: Leveraging Quality Management Tools for MIPS and BeyondHealth Catalyst
Join us as we delve into the crucial realm of quality reporting for MSSP (Medicare Shared Savings Program) Accountable Care Organizations (ACOs).
In this session, we will explore how a robust quality management solution can empower your organization to meet regulatory requirements and improve processes for MIPS reporting and internal quality programs. Learn how our MeasureAble application enables compliance and fosters continuous improvement.
Feeding plate for a newborn with Cleft Palate.pptxSatvikaPrasad
A feeding plate is a prosthetic device used for newborns with a cleft palate to assist in feeding and improve nutrition intake. From a prosthodontic perspective, this plate acts as a barrier between the oral and nasal cavities, facilitating effective sucking and swallowing by providing a more normal anatomical structure. It helps to prevent milk from entering the nasal passage, thereby reducing the risk of aspiration and enhancing the infant's ability to feed efficiently. The feeding plate also aids in the development of the oral muscles and can contribute to better growth and weight gain. Its custom fabrication and proper fitting by a prosthodontist are crucial for ensuring comfort and functionality, as well as for minimizing potential complications. Early intervention with a feeding plate can significantly improve the quality of life for both the infant and the parents.
KEY Points of Leicester travel clinic In London doc.docxNX Healthcare
In order to protect visitors' safety and wellbeing, Travel Clinic Leicester offers a wide range of travel-related health treatments, including individualized counseling and vaccines. Our team of medical experts specializes in getting people ready for international travel, with a particular emphasis on vaccines and health consultations to prevent travel-related illnesses. We provide a range of travel-related services, such as health concerns unique to a trip, prevention of malaria, and travel-related medical supplies. Our clinic is dedicated to providing top-notch care, keeping abreast of the most recent recommendations for vaccinations and travel health precautions. The goal of Travel Clinic Leicester is to keep you safe and well-rested no matter what kind of travel you choose—business, pleasure, or adventure.
PET CT beginners Guide covers some of the underrepresented topics in PET CTMiadAlsulami
This lecture briefly covers some of the underrepresented topics in Molecular imaging with cases , such as:
- Primary pleural tumors and pleural metastases.
- Distinguishing between MPM and Talc Pleurodesis.
- Urological tumors.
- The role of FDG PET in NET.
Hypertension and it's role of physiotherapy in it.Vishal kr Thakur
This particular slides consist of- what is hypertension,what are it's causes and it's effect on body, risk factors, symptoms,complications, diagnosis and role of physiotherapy in it.
This slide is very helpful for physiotherapy students and also for other medical and healthcare students.
Here is summary of hypertension -
Hypertension, also known as high blood pressure, is a serious medical condition that occurs when blood pressure in the body's arteries is consistently too high. Blood pressure is the force of blood pushing against the walls of blood vessels as the heart pumps it. Hypertension can increase the risk of heart disease, brain disease, kidney disease, and premature death.
Hypertension and it's role of physiotherapy in it.
Annual Results and Impact Evaluation Workshop for RBF - Day Two - Measuring Impact - Impact Evaluation Methods for Policy Makers - A Refresher
1. Slides by Sebastian Martinez, Christel Vermeersch and Paul Gertler. We thank Patrick Premand and Martin Ruegenberg for
contributions. The content of this presentation reflects the views of the authors and not necessarily those of the World Bank .
Measuring Impact
Impact Evaluation Methods for
Policy Makers – A Refresher
Gil Shapira
Damien de Walque
2. Our Objective
Estimate the causal effect (impact)
of intervention (P) on outcome (Y).
(P) = Program or Treatment
(Y) = Indicator, Measure of Success
Example: What is the effect of a Cash Transfer Program (P)
on Household Consumption (Y)?
4. Problem of Missing Data
For a program beneficiary:
α= (Y | P=1)-(Y | P=0)
we observe
(Y | P=1): Household Consumption (Y) with
a cash transfer program (P=1)
but we do not observe
(Y | P=0): Household Consumption (Y)
without a cash transfer program (P=0)
6. Estimating impact of P on Y
OBSERVE (Y | P=1)
Outcome with treatment
ESTIMATE (Y | P=0)
The Counterfactual
o Use comparison or
control group
α= (Y | P=1)-(Y | P=0)
IMPACT = - counterfactual
Outcome with
treatment
7. Example: What is the Impact of…
giving Fulanito
(P)
(Y)?
additional pocket money
on Fulanito’s consumption
of candies
9. In reality, use statistics
Treatment Comparison
Average Y=6 candies Average Y=4 Candies
IMPACT=6-4=2 Candies
10. Finding good comparison groups
We want to find clones for the Fulanitos in our
programs.
The treatment and comparison groups should
o have identical characteristics
o except for benefiting from the intervention.
In practice, use program eligibility & assignment
rules to construct valid estimates of the
counterfactuals
11. Case Study: Progresa
National anti-poverty program in Mexico
o Started 1997
o 5 million beneficiaries by 2004
o Eligibility – based on poverty index
Cash Transfers
o Conditional on school and health care attendance.
12. Case Study: Progresa
Rigorous impact evaluation with rich data
o 506 communities, 24,000 households
o Baseline 1997, follow-up 2008
Many outcomes of interest
Here: Consumption per capita
What is the effect of Progresa (P) on
Consumption Per Capita (Y)?
If impact is a increase of $20 or more, then scale up
nationally
16. Case 1: Before & After
What is the effect of Progresa (P) on
consumption (Y)?
Y
TimeT=1997 T=1998
α = $35
IMPACT=A-B= $35
B
A
233
268(1) Observe only
beneficiaries (P=1)
(2) Two observations
in time:
Consumption at T=0
and consumption at
T=1.
17. Case 1: Before & After
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
Consumption (Y)
Outcome with Treatment
(After) 268.7
Counterfactual
(Before) 233.4
Impact
(Y | P=1) - (Y | P=0) 35.3***
18. Case 1: What’s the problem?
Y
TimeT=0 T=1
α = $35
B
A
233
268
Economic Boom:
o Real Impact=A-C
o A-B is an
overestimate
C ?
D ?
Impact?
Impact?
Economic Recession:
o Real Impact=A-D
o A-B is an
underestimate
20. False Counterfactual #2
If we have post-treatment data on
o Enrolled: treatment group
o Not-enrolled: “control” group (counterfactual)
Those ineligible to participate.
Or those that choose NOT to participate.
Selection Bias
o Reason for not enrolling may be correlated
with outcome (Y)
Control for observables.
But not un-observables!
o Estimated impact is confounded with other
things.
Enrolled & Not Enrolled
21. Measure outcomes in post-treatment (T=1)
Case 2: Enrolled & Not Enrolled
Enrolled
Y=268
Not Enrolled
Y=290
Ineligibles
(Non-Poor)
Eligibles
(Poor)
In what ways might E&NE be different, other than their enrollment in the program?
22. Case 2: Enrolled & Not Enrolled
Consumption (Y)
Outcome with Treatment
(Enrolled) 268
Counterfactual
(Not Enrolled) 290
Impact
(Y | P=1) - (Y | P=0) -22**
Estimated Impact on
Consumption (Y)
Linear Regression -22**
Multivariate Linear
Regression -4.15
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
23. Progresa Policy Recommendation?
Will you recommend scaling up Progresa?
B&A: Are there other time-varying factors that also
influence consumption?
E&NE:
o Are reasons for enrolling correlated with consumption?
o Selection Bias.
Impact on Consumption (Y)
Case 1: Before & After 35.3***
Case 2: Enrolled& Not Enrolled -22**
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
24. B&A
Compare: Same individuals
Before and After they
receive P.
Problem: Other things may
have happened over time.
E&NE
Compare: Group of
individuals Enrolled in a
program with group that
chooses not to enroll.
Problem: Selection Bias.
We don’t know why they
are not enrolled.
Keep in Mind
Both counterfactuals may
lead to biased estimates of
the counterfactual and the
impact.
!
26. Randomized Treatments & Controls
o Randomize!
o Lottery for who is offered benefits
o Fair, transparent and ethical way to assign benefits to equally
deserving populations.
Eligibles > Number of Benefits
o Give each eligible unit the same chance of receiving treatment
o Compare those offered treatment with those not offered
treatment (controls).
Oversubscription
o Give each eligible unit the same chance of receiving treatment
first, second, third…
o Compare those offered treatment first, with those
offered later (controls).
Randomized Phase In
28. Case 3: Randomized Assignment
Progresa CCT program
Unit of randomization: Community
o 320 treatment communities (14446 households):
First transfers in April 1998.
o 186 control communities (9630 households):
First transfers November 1999
506 communities in the evaluation sample
Randomized phase-in
29. Case 3: Randomized Assignment
Treatment
Communities
320
Control
Communities
186
Time
T=1T=0
Comparison Period
30. Case 3: Randomized Assignment
How do we know we have
good clones?
In the absence of Progresa, treatment
and comparisons should be identical
Let’s compare their characteristics at
baseline (T=0)
31. Case 3: Balance at Baseline
Case 3: Randomized Assignment
Control Treatment T-stat
Consumption
($ monthly per capita) 233.47 233.4 -0.39
Head’s age
(years) 42.3 41.6 1.2
Spouse’s age
(years) 36.8 36.8 -0.38
Head’s education
(years) 2.8 2.9 -2.16**
Spouse’s education
(years) 2.6 2.7 -0.006
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
32. Case 3: Balance at Baseline
Case 3: Randomized Assignment
Control Treatment T-stat
Head is female=1 0.07 0.07 0.66
Indigenous=1 0.42 0.42 0.21
Number of household
members 5.7 5.7 -1.21
Bathroom=1 0.56 0.57 -1.04
Hectares of Land 1.71 1.67 1.35
Distance to Hospital
(km) 106 109 -1.02
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
33. Case 3: Randomized Assignment
Treatment Group
(Randomized to
treatment)
Counterfactual
(Randomized to
Comparison)
Impact
(Y | P=1) - (Y | P=0)
Baseline (T=0)
Consumption (Y) 233.47 233.40 0.07
Follow-up (T=1)
Consumption (Y) 268.75 239.5 29.25**
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
34. Progresa Policy Recommendation?
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
Impact of Progresa on Consumption (Y)
Case 1: Before & After 35.27**
Case 2: Enrolled& Not Enrolled -22**
Case 3: Randomized Assignment 29.25**
35. Keep in Mind
Randomized Assignment
In Randomized Assignment,
large enough samples,
produces 2 statistically
equivalent groups.
We have identified the
perfect clone.
Randomized
beneficiary
Randomized
comparison
Feasible for prospective
evaluations with over-
subscription/excess demand.
Most pilots and new
programs fall into this
category.
!
36. Randomized assignment with
different benefit levels
Traditional impact evaluation question:
o What is the impact of a program on an outcome?
Other policy question of interest:
o What is the optimal level for program benefits?
o What is the impact of a “higher-intensity” treatment
compared to a “lower-intensity” treatment?
Randomized assignment with 2 levels of benefits:
Comparison Low Benefit High Benefit
X
37. = Ineligible
Randomized assignment with
different benefit levels
= Eligible
1. Eligible Population 2. Evaluation sample
3. Randomize
treatment
(2 benefit levels)
Comparison
43. Case 6: Difference in differences
Enrolled Not Enrolled Difference
Baseline (T=0)
Consumption (Y) 233.47 281.74 -48.27
Follow-up (T=1)
Consumption (Y) 268.75 290 -21.25
Difference 35.28 8.26 27.02
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
44. Progresa Policy Recommendation?
Note: If the effect is statistically significant at the 1% significance level, we label the
estimated impact with 2 stars (**).
Impact of Progresa on Consumption (Y)
Case 1: Before & After 35.27**
Case 2: Enrolled & Not Enrolled -22**
Case 3: Randomized Assignment 29.25**
Case 6: Difference-in-Differences 27.02**
45. Keep in Mind
Difference-in-Differences
Differences in Differences
combines Enrolled & Not
Enrolled with Before & After.
Slope: Generate
counterfactual for change in
outcome
Trends –slopes- are the same
in treatments and controls
(Fundamental assumption).
To test this, at least 3
observations in time are
needed:
o 2 observations before
o 1 observation after.
!
47. Remember
The objective of impact evaluation
is to estimate the causal effect or
impact of a program on outcomes
of interest.
48. Remember
To estimate impact, we need to
estimate the counterfactual.
o what would have happened in the absence of
the program and
o use comparison or control groups.
49. Remember
Choose the best evaluation
method that is feasible in the
program’s operational context.
50. Reference
This material constitutes supporting material for the "Impact Evaluation in
Practice book. This additional material is made freely but please
acknowledge its use as follows:
Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J.
Vermeersch, 2010, Impact Evaluation in Practice: Ancillary
Material, The World Bank, Washington DC
(www.worldbank.org/ieinpractice).
The content of this presentation reflects the views of the authors and not
necessarily those of the World Bank."