Causal Inference Opening Workshop - Assisting the Impact of State Polcies: Brdiging the Gap Between Theory & Practice - Beth Ann Griffin, December 10, 2019
This document summarizes a simulation study evaluating causal inference methods for assessing the effects of opioid and gun policies. The study used real US state-level data to simulate the adoption of policies by some states and estimated the effects using different statistical models. It found that with fewer adopting states, type 1 error rates were too high, and most models lacked power. It recommends using cluster-robust standard errors and lagged outcomes to improve model performance. The study aims to help identify best practices for policy evaluation studies.
Ragui Assaad- University of Minnesota
Caroline Krafft- ST. Catherine University
ERF Training on Applied Micro-Econometrics and Public Policy Evaluation
Cairo, Egypt July 25-27, 2016
www.erf.org.eg
International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.
Recently, the machine learning community has expressed strong interest in applying latent variable modeling strategies to causal inference problems with unobserved confounding. Here, I discuss one of the big debates that occurred over the past year, and how we can move forward. I will focus specifically on the failure of point identification in this setting, and discuss how this can be used to design flexible sensitivity analyses that cleanly separate identified and unidentified components of the causal model.
I will discuss paradigmatic statistical models of inference and learning from high dimensional data, such as sparse PCA and the perceptron neural network, in the sub-linear sparsity regime. In this limit the underlying hidden signal, i.e., the low-rank matrix in PCA or the neural network weights, has a number of non-zero components that scales sub-linearly with the total dimension of the vector. I will provide explicit low-dimensional variational formulas for the asymptotic mutual information between the signal and the data in suitable sparse limits. In the setting of support recovery these formulas imply sharp 0-1 phase transitions for the asymptotic minimum mean-square-error (or generalization error in the neural network setting). A similar phase transition was analyzed recently in the context of sparse high-dimensional linear regression by Reeves et al.
More Related Content
Similar to Causal Inference Opening Workshop - Assisting the Impact of State Polcies: Brdiging the Gap Between Theory & Practice - Beth Ann Griffin, December 10, 2019
Ragui Assaad- University of Minnesota
Caroline Krafft- ST. Catherine University
ERF Training on Applied Micro-Econometrics and Public Policy Evaluation
Cairo, Egypt July 25-27, 2016
www.erf.org.eg
International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.
Repeated-Measures and Two-Factor Analysis of Variancejasondroesch
Chapter 13 Power Points for Essentials of Statistics for the Behavioral Sciences, Gravetter & Wallnau, 8th ed
Similar to Causal Inference Opening Workshop - Assisting the Impact of State Polcies: Brdiging the Gap Between Theory & Practice - Beth Ann Griffin, December 10, 2019 (20)
Recently, the machine learning community has expressed strong interest in applying latent variable modeling strategies to causal inference problems with unobserved confounding. Here, I discuss one of the big debates that occurred over the past year, and how we can move forward. I will focus specifically on the failure of point identification in this setting, and discuss how this can be used to design flexible sensitivity analyses that cleanly separate identified and unidentified components of the causal model.
I will discuss paradigmatic statistical models of inference and learning from high dimensional data, such as sparse PCA and the perceptron neural network, in the sub-linear sparsity regime. In this limit the underlying hidden signal, i.e., the low-rank matrix in PCA or the neural network weights, has a number of non-zero components that scales sub-linearly with the total dimension of the vector. I will provide explicit low-dimensional variational formulas for the asymptotic mutual information between the signal and the data in suitable sparse limits. In the setting of support recovery these formulas imply sharp 0-1 phase transitions for the asymptotic minimum mean-square-error (or generalization error in the neural network setting). A similar phase transition was analyzed recently in the context of sparse high-dimensional linear regression by Reeves et al.
Many different measurement techniques are used to record neural activity in the brains of different organisms, including fMRI, EEG, MEG, lightsheet microscopy and direct recordings with electrodes. Each of these measurement modes have their advantages and disadvantages concerning the resolution of the data in space and time, the directness of measurement of the neural activity and which organisms they can be applied to. For some of these modes and for some organisms, significant amounts of data are now available in large standardized open-source datasets. I will report on our efforts to apply causal discovery algorithms to, among others, fMRI data from the Human Connectome Project, and to lightsheet microscopy data from zebrafish larvae. In particular, I will focus on the challenges we have faced both in terms of the nature of the data and the computational features of the discovery algorithms, as well as the modeling of experimental interventions.
Bayesian Additive Regression Trees (BART) has been shown to be an effective framework for modeling nonlinear regression functions, with strong predictive performance in a variety of contexts. The BART prior over a regression function is defined by independent prior distributions on tree structure and leaf or end-node parameters. In observational data settings, Bayesian Causal Forests (BCF) has successfully adapted BART for estimating heterogeneous treatment effects, particularly in cases where standard methods yield biased estimates due to strong confounding.
We introduce BART with Targeted Smoothing, an extension which induces smoothness over a single covariate by replacing independent Gaussian leaf priors with smooth functions. We then introduce a new version of the Bayesian Causal Forest prior, which incorporates targeted smoothing for modeling heterogeneous treatment effects which vary smoothly over a target covariate. We demonstrate the utility of this approach by applying our model to a timely women's health and policy problem: comparing two dosing regimens for an early medical abortion protocol, where the outcome of interest is the probability of a successful early medical abortion procedure at varying gestational ages, conditional on patient covariates. We discuss the benefits of this approach in other women’s health and obstetrics modeling problems where gestational age is a typical covariate.
Difference-in-differences is a widely used evaluation strategy that draws causal inference from observational panel data. Its causal identification relies on the assumption of parallel trends, which is scale-dependent and may be questionable in some applications. A common alternative is a regression model that adjusts for the lagged dependent variable, which rests on the assumption of ignorability conditional on past outcomes. In the context of linear models, Angrist and Pischke (2009) show that the difference-in-differences and lagged-dependent-variable regression estimates have a bracketing relationship. Namely, for a true positive effect, if ignorability is correct, then mistakenly assuming parallel trends will overestimate the effect; in contrast, if the parallel trends assumption is correct, then mistakenly assuming ignorability will underestimate the effect. We show that the same bracketing relationship holds in general nonparametric (model-free) settings. We also extend the result to semiparametric estimation based on inverse probability weighting.
We develop sensitivity analyses for weak nulls in matched observational studies while allowing unit-level treatment effects to vary. In contrast to randomized experiments and paired observational studies, we show for general matched designs that over a large class of test statistics, any valid sensitivity analysis for the weak null must be unnecessarily conservative if Fisher's sharp null of no treatment effect for any individual also holds. We present a sensitivity analysis valid for the weak null, and illustrate why it is conservative if the sharp null holds through connections to inverse probability weighted estimators. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and is valid for the weak null under additional assumptions which may be deemed reasonable by practitioners. The methods may be applied to matched observational studies constructed using any optimal without-replacement matching algorithm, allowing practitioners to assess robustness to hidden bias while allowing for treatment effect heterogeneity.
The world of health care is full of policy interventions: a state expands eligibility rules for its Medicaid program, a medical society changes its recommendations for screening frequency, a hospital implements a new care coordination program. After a policy change, we often want to know, “Did it work?” This is a causal question; we want to know whether the policy CAUSED outcomes to change. One popular way of estimating causal effects of policy interventions is a difference-in-differences study. In this controlled pre-post design, we measure the change in outcomes of people who are exposed to the new policy, comparing average outcomes before and after the policy is implemented. We contrast that change to the change over the same time period in people who were not exposed to the new policy. The differential change in the treated group’s outcomes, compared to the change in the comparison group’s outcomes, may be interpreted as the causal effect of the policy. To do so, we must assume that the comparison group’s outcome change is a good proxy for the treated group’s (counterfactual) outcome change in the absence of the policy. This conceptual simplicity and wide applicability in policy settings makes difference-in-differences an appealing study design. However, the apparent simplicity belies a thicket of conceptual, causal, and statistical complexity. In this talk, I will introduce the fundamentals of difference-in-differences studies and discuss recent innovations including key assumptions and ways to assess their plausibility, estimation, inference, and robustness checks.
We present recent advances and statistical developments for evaluating Dynamic Treatment Regimes (DTR), which allow the treatment to be dynamically tailored according to evolving subject-level data. Identification of an optimal DTR is a key component for precision medicine and personalized health care. Specific topics covered in this talk include several recent projects with robust and flexible methods developed for the above research area. We will first introduce a dynamic statistical learning method, adaptive contrast weighted learning (ACWL), which combines doubly robust semiparametric regression estimators with flexible machine learning methods. We will further develop a tree-based reinforcement learning (T-RL) method, which builds an unsupervised decision tree that maintains the nature of batch-mode reinforcement learning. Unlike ACWL, T-RL handles the optimization problem with multiple treatment comparisons directly through a purity measure constructed with augmented inverse probability weighted estimators. T-RL is robust, efficient and easy to interpret for the identification of optimal DTRs. However, ACWL seems more robust against tree-type misspecification than T-RL when the true optimal DTR is non-tree-type. At the end of this talk, we will also present a new Stochastic-Tree Search method called ST-RL for evaluating optimal DTRs.
A fundamental feature of evaluating causal health effects of air quality regulations is that air pollution moves through space, rendering health outcomes at a particular population location dependent upon regulatory actions taken at multiple, possibly distant, pollution sources. Motivated by studies of the public-health impacts of power plant regulations in the U.S., this talk introduces the novel setting of bipartite causal inference with interference, which arises when 1) treatments are defined on observational units that are distinct from those at which outcomes are measured and 2) there is interference between units in the sense that outcomes for some units depend on the treatments assigned to many other units. Interference in this setting arises due to complex exposure patterns dictated by physical-chemical atmospheric processes of pollution transport, with intervention effects framed as propagating across a bipartite network of power plants and residential zip codes. New causal estimands are introduced for the bipartite setting, along with an estimation approach based on generalized propensity scores for treatments on a network. The new methods are deployed to estimate how emission-reduction technologies implemented at coal-fired power plants causally affect health outcomes among Medicare beneficiaries in the U.S.
Laine Thomas presented information about how causal inference is being used to determine the cost/benefit of the two most common surgical surgical treatments for women - hysterectomy and myomectomy.
We provide an overview of some recent developments in machine learning tools for dynamic treatment regime discovery in precision medicine. The first development is a new off-policy reinforcement learning tool for continual learning in mobile health to enable patients with type 1 diabetes to exercise safely. The second development is a new inverse reinforcement learning tools which enables use of observational data to learn how clinicians balance competing priorities for treating depression and mania in patients with bipolar disorder. Both practical and technical challenges are discussed.
The method of differences-in-differences (DID) is widely used to estimate causal effects. The primary advantage of DID is that it can account for time-invariant bias from unobserved confounders. However, the standard DID estimator will be biased if there is an interaction between history in the after period and the groups. That is, bias will be present if an event besides the treatment occurs at the same time and affects the treated group in a differential fashion. We present a method of bounds based on DID that accounts for an unmeasured confounder that has a differential effect in the post-treatment time period. These DID bracketing bounds are simple to implement and only require partitioning the controls into two separate groups. We also develop two key extensions for DID bracketing bounds. First, we develop a new falsification test to probe the key assumption that is necessary for the bounds estimator to provide consistent estimates of the treatment effect. Next, we develop a method of sensitivity analysis that adjusts the bounds for possible bias based on differences between the treated and control units from the pretreatment period. We apply these DID bracketing bounds and the new methods we develop to an application on the effect of voter identification laws on turnout. Specifically, we focus estimating whether the enactment of voter identification laws in Georgia and Indiana had an effect on voter turnout.
We study experimental design in large-scale stochastic systems with substantial uncertainty and structured cross-unit interference. We consider the problem of a platform that seeks to optimize supply-side payments p in a centralized marketplace where different suppliers interact via their effects on the overall supply-demand equilibrium, and propose a class of local experimentation schemes that can be used to optimize these payments without perturbing the overall market equilibrium. We show that, as the system size grows, our scheme can estimate the gradient of the platform’s utility with respect to p while perturbing the overall market equilibrium by only a vanishingly small amount. We can then use these gradient estimates to optimize p via any stochastic first-order optimization method. These results stem from the insight that, while the system involves a large number of interacting units, any interference can only be channeled through a small number of key statistics, and this structure allows us to accurately predict feedback effects that arise from global system changes using only information collected while remaining in equilibrium.
We discuss a general roadmap for generating causal inference based on observational studies used to general real world evidence. We review targeted minimum loss estimation (TMLE), which provides a general template for the construction of asymptotically efficient plug-in estimators of a target estimand for realistic (i.e, infinite dimensional) statistical models. TMLE is a two stage procedure that first involves using ensemble machine learning termed super-learning to estimate the relevant stochastic relations between the treatment, censoring, covariates and outcome of interest. The super-learner allows one to fully utilize all the advances in machine learning (in addition to more conventional parametric model based estimators) to build a single most powerful ensemble machine learning algorithm. We present Highly Adaptive Lasso as an important machine learning algorithm to include.
In the second step, the TMLE involves maximizing a parametric likelihood along a so-called least favorable parametric model through the super-learner fit of the relevant stochastic relations in the observed data. This second step bridges the state of the art in machine learning to estimators of target estimands for which statistical inference is available (i.e, confidence intervals, p-values etc). We also review recent advances in collaborative TMLE in which the fit of the treatment and censoring mechanism is tailored w.r.t. performance of TMLE. We also discuss asymptotically valid bootstrap based inference. Simulations and data analyses are provided as demonstrations.
We describe different approaches for specifying models and prior distributions for estimating heterogeneous treatment effects using Bayesian nonparametric models. We make an affirmative case for direct, informative (or partially informative) prior distributions on heterogeneous treatment effects, especially when treatment effect size and treatment effect variation is small relative to other sources of variability. We also consider how to provide scientifically meaningful summaries of complicated, high-dimensional posterior distributions over heterogeneous treatment effects with appropriate measures of uncertainty.
Climate change mitigation has traditionally been analyzed as some version of a public goods game (PGG) in which a group is most successful if everybody contributes, but players are best off individually by not contributing anything (i.e., “free-riding”)—thereby creating a social dilemma. Analysis of climate change using the PGG and its variants has helped explain why global cooperation on GHG reductions is so difficult, as nations have an incentive to free-ride on the reductions of others. Rather than inspire collective action, it seems that the lack of progress in addressing the climate crisis is driving the search for a “quick fix” technological solution that circumvents the need for cooperation.
This seminar discussed ways in which to produce professional academic writing, from academic papers to research proposals or technical writing in general.
Machine learning (including deep and reinforcement learning) and blockchain are two of the most noticeable technologies in recent years. The first one is the foundation of artificial intelligence and big data, and the second one has significantly disrupted the financial industry. Both technologies are data-driven, and thus there are rapidly growing interests in integrating them for more secure and efficient data sharing and analysis. In this paper, we review the research on combining blockchain and machine learning technologies and demonstrate that they can collaborate efficiently and effectively. In the end, we point out some future directions and expect more researches on deeper integration of the two promising technologies.
In this talk, we discuss QuTrack, a Blockchain-based approach to track experiment and model changes primarily for AI and ML models. In addition, we discuss how change analytics can be used for process improvement and to enhance the model development and deployment processes.
This talk builds on recent empirical work addressing the extent to which the transaction graph serves as an early-warning indicator for large financial losses. By identifying certain sub-graphs ('chainlets') with causal effect on price movements, we demonstrate the impact of extreme transaction graph activity on the intraday volatility of the Bitcoin prices series. In particular, we infer the loss distributions conditional on extreme chainlet activity. Armed with this empirical representation, we propose a modeling approach to explore conditions under which the market is stabilized by transaction graph aware agents.
More from The Statistical and Applied Mathematical Sciences Institute (20)
This is a presentation by Dada Robert in a Your Skill Boost masterclass organised by the Excellence Foundation for South Sudan (EFSS) on Saturday, the 25th and Sunday, the 26th of May 2024.
He discussed the concept of quality improvement, emphasizing its applicability to various aspects of life, including personal, project, and program improvements. He defined quality as doing the right thing at the right time in the right way to achieve the best possible results and discussed the concept of the "gap" between what we know and what we do, and how this gap represents the areas we need to improve. He explained the scientific approach to quality improvement, which involves systematic performance analysis, testing and learning, and implementing change ideas. He also highlighted the importance of client focus and a team approach to quality improvement.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
Ethnobotany and Ethnopharmacology:
Ethnobotany in herbal drug evaluation,
Impact of Ethnobotany in traditional medicine,
New development in herbals,
Bio-prospecting tools for drug discovery,
Role of Ethnopharmacology in drug evaluation,
Reverse Pharmacology.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
Palestine last event orientationfvgnh .pptxRaedMohamed3
An EFL lesson about the current events in Palestine. It is intended to be for intermediate students who wish to increase their listening skills through a short lesson in power point.
Home assignment II on Spectroscopy 2024 Answers.pdf
Causal Inference Opening Workshop - Assisting the Impact of State Polcies: Brdiging the Gap Between Theory & Practice - Beth Ann Griffin, December 10, 2019
2. • Grateful for support from the
National Institute on Drug Abuse of
the NIH and Arnold Ventures
• All views and opinions expressed
are my own
• No conflicts of interest
• Colleagues I would like to acknowledge
• Elizabeth Stuart, PhD
• Megan Schuler, PhD
• Andrew Morral, PhD
• Terry Schell, PhD
• Rosalie Liccardo Pacula, PhD
• Bradley D. Stein, MD, PhD
• Mary Vaiana, PhD
• Stephen Patrick, MD PhD
• Elizabeth McNeer, MS
• Rosanna Smart, PhD
• David Powell, PhD
• Matt Cefalu, PhD
3. • Why assessing state opioid and gun policies is so challenging
• Our approach to evaluate performance of methods
• Key simulation results & their implications for practice
• Need for new methods & better dissemination of new methods
4. • Selection bias: States that choose to adopt policies may differ from
states that do not
• Sparse data: Policies may be adopted by a limited number of states,
or adopted recently (limited number of post-policy observations)
• Policy heterogeneity: States may implement related, but distinct,
versions of a specific policy
• Concurrent policies: States may have multiple policies targeting
opioids / guns
5. • Use counterfactual approach:
Compare what actually happened
in a state with what we estimate
would have happened without the
policy change
7. • 85% longitudinal design vs 15% cross-sectional design
• 70% multi-state design vs 30% single-state design
• Multi-state longitudinal studies (n=79):
• 45% Difference-in-difference design
• 30% Comparative interrupted time series
• 15% GEE
8. • Strategy:
• Use simulation studies on existing data to identify the best causal
inference methods for assessing the effects of opioid and gun policies
• Tactic:
• Using existing data (state-level for now), simulate the effect of a policy
on outcomes, and determine which method(s) most accurately detect
the effect
9. • Longitudinal, repeated measures data collected on an annual basis
• States are the units of interest, observations are clustered within state
• Analyses comprised all 50 states
• Staggered implementation of given policy across “treatment” states
10. Simulate policy
effect in real
data
Estimate effect
with statistical
models
Compare
model
performance
Generated simulated
dataset by augmenting
real state-level data;
N = 50 states
Compared 17 different
statistical models under
multiple conditions (5,000
replications each)
Four performance
measures: Type 1 error
rates (“false positives”),
power, bias, root mean
squared error (RMSE)
14. Opioid Deaths
(per 100,000
population)
Simulate Estimate Compare
1. Real U.S. state opioid-related death rates
2. Randomly select 5 states
3. Randomly select policy implementation
date
4. Introduce policy effect after
implementation date
15. GLM
Regression
specification Weighting Standard error adj
1 Linear Fixed effects (FE) Population Weighted None, Huber, Cluster
2 Unweighted None, Huber, Cluster
3 FE + Detrended Population Weighted None, Huber, Cluster
4 Unweighted None, Huber, Cluster
5 Autoregressive Population Weighted None, Huber, Cluster
6 Unweighted None, Huber, Cluster
7 GEE model Population Weighted NA
8 Unweighted NA
9 Log-linear Fixed effects (FE) Population Weighted None, Huber, Cluster
10 Unweighted None, Huber, Cluster
11 Autoregressive Population Weighted None, Huber, Cluster
12 Unweighted None, Huber, Cluster
13
Negative
Binomial
Fixed effects (FE) Unweighted None, Huber, Cluster
14 FE + Detrended Unweighted None, Huber, Cluster
15 Autoregressive Unweighted None, Huber, Cluster
16 Poisson Fixed effects (FE) Unweighted None, Huber, Cluster
17 Autoregressive Unweighted None, Huber, Cluster
16.
17. Simulation Project
0
0.1
0.2
0.3
0.4
0.5
No Adjustment Huber Cluster
1 5 15 30
Goal: Models
with Type 1
error = 0.05
Type 1
Error Rate
for Linear
Two-Way
Fixed Effects
Model with
Population
Weights
Number of States Implementing Policy
19. Simulation Project
0.00 0.05 0.10 0.15 0.20 0.25
Linear 2-way FE Wted
Linear GEE Wted
Log Y 2-way FE Wted
Linear 2-way FE Unwt
Linear Detrended Wted
Poisson 2-way FE
Linear Detrended Unwt
Log Y AR Unwt
Log Y AR Wted
Poisson AR
Linear GEE Unwt
Log Y 2-way FE Unwt
Negative Binomial 2-way FE
Negative Binomial AR
Linear AR Unwt
Negative Binomial Detrended
Linear AR Wted
Power
21. • Type I error rates are unreasonably high when number of states
implementing new policy is low (< 15)
• Caution needed when such studies report “statistically significant”
findings - may be a false positive (e.g., saying a law as an effect when it
truly does not)
• Critical to use cluster adjustments to standard errors when using state
and year fixed effects in linear or log-linear models
• Recommend use of a correction factor to ensure Type I error = 0.05
• Power is very low for all models; need to find better approaches to account
for level of uncertainty
• Use of lagged outcomes as control covariates is helpful in the linear
model
• Use of negative binomial link performs better than Poisson
22. • State-policy evaluations need new methods
• Many state-of-the-art methods coming from statisticians, economists
and other methodologists
• Application of new methods typically lags their development
• Need more effective dissemination strategies to get new methods
into the hands of the broader scientific community (not just
methodologists)
• Use Shiny to help, teach courses/workshops at conferences, create
websites, actively disseminate your work
23. • Understanding how theory performs in practice is essential
• Room for methods development where applications are at the core
• Annals of Applied Statistics (AOAS) – perfect home for papers tackling
these issues – we want
• Papers that include innovative methodology brought to bear on
scientific/policy questions and relevant data
• Groundbreaking application of state-of-the-art methods
26. Difference in difference (DID) method
Classic 2-way fixed effects DID specification:
𝑔(𝑌𝑖𝑡) = 𝛼 ∙ 𝐴𝑖𝑡 + 𝜷 ∙ 𝑿𝑖𝑡 + 𝜌𝒊 + 𝜎𝑡 + 𝜀𝑖𝑡
• State fixed effects (𝜌𝑖): baseline differences across states
• Time fixed effects (𝜎𝑡): temporal national trends
Detrended model
• Include state-specific linear trends
27. Autoregressive Model
• Include one-period lagged autoregressive (AR) model
Generalized Estimating Equation (GEE) Model
• Semi-parametric method that requires specification of the covariance
matrix for within-subject observations
28.
29. Simulation Project
Bias = Total Count of Deaths by which model over or estimated effect when true effect = 5% (or 700 deaths)
-100.00 0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00
Linear AR; Population Wted
Linear AR; Unwted
Linear Detrended; Population Wted
Linear Two-Way FE; Unwted
Linear GEE; Weighhted
Linear Two-Way FE; Population Wted
Linear GEE; Unweighted
Linear Detrended; Unwted
Log-Y Two-Way FE; Unwted
Negative Binomial; Two-Way FE
Negative Binomial; Detrended
Poisson; Two-Way FE
Log-Y Two-Way FE; Population Wted
Negative Binomial; AR
Log-Y AR; Population Wted
Log-Y AR; Unwted
Poisson; AR
30. 0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00
Linear GEE; Weighhted
Linear Two-Way FE; Population Wted
Linear Detrended; Population Wted
Linear Two-Way FE; Unwted
Linear Detrended; Unwted
Linear GEE; Unweighted
Linear AR; Unwted
Linear AR; Population Wted
Poisson; AR
Log-Y AR; Population Wted
Log-Y AR; Unwted
Negative Binomial; AR
Log-Y Two-Way FE; Population Wted
Poisson; Two-Way FE
Negative Binomial; Detrended
Log-Y Two-Way FE; Unwted
Negative Binomial; Two-Way FE
Simulation Project
Root Mean Square Error in Model where True Policy Effect = 0
33. Inclusion criteria:
1) Must be study estimating the impact of a relevant policy on
opioid-related outcomes
2) Restricted focus to state or federal level policies
• excluded local, hospital-level initiatives
3) Published during 2005 – 2018
4) U.S. studies only
34. • Literature review entailed structured extraction regarding details of
study population, study period, analytical design, data sources, etc.
• 146 studies met inclusion criteria
• Created taxonomy to systematically classify:
• Opioid-related policies
• Opioid-related outcomes