Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
New Hypothesis Testing Method Guy Lion November 2009
Challenges <ul><li>Colleagues raised the following challenges within hypothesis testing: </li></ul><ul><li>You don’t have ...
An inadequate solution: Paired testing  So what can we do? <ul><li>If you have no controls, you could conduct paired testi...
Combining Regression and Effect Size <ul><li>Let’s consider a situation where you have a single store and no Controls.  </...
Structure of the Regression <ul><li>The regression includes:  </li></ul><ul><li>A trend variable that captures by how much...
Developing the Regression and  interpreting the Effect Size <ul><li>First, you calculate the regressed Y estimates to cons...
Building the model I am measuring a variable Y that trended upward from 80 in January 2007 to 95 in August 2009.  The most...
Quantifying the Effect Size The Effect Size of the treatment for the first month (June 2009) is equal to the regressed coe...
Advantages of this method <ul><li>It solves the three challenges expressed on slide 2.  </li></ul><ul><li>It generates Eff...
Technical Appendix
Autocorrelation of residuals Time series often break a linear regression assumption that residuals are independent when th...
Heteroskedasticity Linear regression assumes the variance of the errors remains constant (homoskedasticity).  Time series ...
Upcoming SlideShare
Loading in …5
×

New Hypothesis Testing Method

1,643 views

Published on

This is a new hypothesis testing method that allows you to test whether something during a "treatment period" made a difference even in the absence of Controls. Even a sample of 1 with no Controls will work.

  • Be the first to comment

New Hypothesis Testing Method

  1. 1. New Hypothesis Testing Method Guy Lion November 2009
  2. 2. Challenges <ul><li>Colleagues raised the following challenges within hypothesis testing: </li></ul><ul><li>You don’t have a Control Group. </li></ul><ul><li>Your sample is too small. </li></ul><ul><li>You are more concerned about the volatility within the history of each subject then between subjects… </li></ul>
  3. 3. An inadequate solution: Paired testing So what can we do? <ul><li>If you have no controls, you could conduct paired testing. But, this is inadequate because you don’t know if any improvement between the pre and post periods is real. This is because of three sources of noise: </li></ul><ul><li>Trend (change over time); </li></ul><ul><li>Seasonality; and </li></ul><ul><li>Change in business conditions. </li></ul>
  4. 4. Combining Regression and Effect Size <ul><li>Let’s consider a situation where you have a single store and no Controls. </li></ul><ul><li>You just have the store monthly historical data. </li></ul><ul><li>In the most recent three months, you apply a “treatment” and you want to check if it made a difference. Can you do it? </li></ul><ul><li>Yes… with a regression and interpreting the results as Effect Size. </li></ul>
  5. 5. Structure of the Regression <ul><li>The regression includes: </li></ul><ul><li>A trend variable that captures by how much Y increases over time; </li></ul><ul><li>Seasonality (as a qualitative variable in XLStat or 11 dummy variables in Excel); </li></ul><ul><li>A dummy variable for each of the “treated” month; </li></ul>If needed, we could add a macroeconomic variable to capture change in the business environment.
  6. 6. Developing the Regression and interpreting the Effect Size <ul><li>First, you calculate the regressed Y estimates to construct its historical trend. </li></ul><ul><li>Second, you calculate a second set of Y estimates but without the “treatment” period dummy variables. </li></ul><ul><li>Third, you observe the Effect Size as the difference between Actuals minus Y estimates without the dummy variables. </li></ul>
  7. 7. Building the model I am measuring a variable Y that trended upward from 80 in January 2007 to 95 in August 2009. The most recent three months, to the right of the red dotted line, consist of the “treatment” period. The latter had an effect. And, its “Effect Size” is growing as the store is performing above trend.
  8. 8. Quantifying the Effect Size The Effect Size of the treatment for the first month (June 2009) is equal to the regressed coefficient 0.74, as the treatment bumped the Y value by that much. Using the standard error of the coefficient, the output builds a 95% confidence interval (0.42 to 1.05). The p value of this coefficient is 0%, so we are nearly 100% confident that this coefficient is different from zero.
  9. 9. Advantages of this method <ul><li>It solves the three challenges expressed on slide 2. </li></ul><ul><li>It generates Effect Size and confidence interval info. </li></ul><ul><li>It calculates p values similar to the ones generated in a regular hypothesis test. </li></ul><ul><li>It conducts a test for each month of the “treatment” period reducing the probability that a single test result is erroneous. </li></ul>
  10. 10. Technical Appendix
  11. 11. Autocorrelation of residuals Time series often break a linear regression assumption that residuals are independent when they are not. Large positive errors follow large positive ones. And, large negative errors follow large negative ones. Autocorrelation of residuals does not affect the regression coefficients. But, it affects the standard error and confidence intervals around such coefficients. To fix this problem use the lagged (t-1) residual of the original regression as an extra variable in a second regression. This example does not exhibit autocorrelation of residuals since the Durbin Watson score is close to 2.0. The seasonality variable and the focus on dummy variables reduce the risk of running into this problem.
  12. 12. Heteroskedasticity Linear regression assumes the variance of the errors remains constant (homoskedasticity). Time series often exhibit a change in the variance of errors (heteroskedasticity). Just like autocorrelation of residuals, heteroskedasticity does not affect the regression coefficients. But, it affects the standard error and confidence intervals around the coefficients. To fix this issue, transform the Y variable into a % change or log. This example does not demonstrate heteroskedasticity. The trend variable and the focus on dummy variables reduce the risk of running into this situation. Also, the Y variable being leveled (it increases by only 20% from beginning to end of series) is unlikely to be heteroskedastic.

×