The document discusses a new hypothesis testing method that addresses challenges in hypothesis testing, particularly the absence of control groups and small sample sizes. It proposes using regression analysis to measure the effect size of treatments applied in a single store, incorporating factors such as trend, seasonality, and macroeconomic variables. Additionally, it highlights the advantages of this method, including generating p-values and confidence intervals, while also addressing issues related to autocorrelation and heteroskedasticity in regression analysis.

1. New Hypothesis Testing Method Guy Lion November 2009

2. Challenges Colleagues raised the following challenges within hypothesis testing: You don’t have a Control Group. Your sample is too small. You are more concerned about the volatility within the history of each subject then between subjects…

3. An inadequate solution: Paired testing So what can we do? If you have no controls, you could conduct paired testing. But, this is inadequate because you don’t know if any improvement between the pre and post periods is real. This is because of three sources of noise: Trend (change over time); Seasonality; and Change in business conditions.

4. Combining Regression and Effect Size Let’s consider a situation where you have a single store and no Controls. You just have the store monthly historical data. In the most recent three months, you apply a “treatment” and you want to check if it made a difference. Can you do it? Yes… with a regression and interpreting the results as Effect Size.

5. Structure of the Regression The regression includes: A trend variable that captures by how much Y increases over time; Seasonality (as a qualitative variable in XLStat or 11 dummy variables in Excel); A dummy variable for each of the “treated” month; If needed, we could add a macroeconomic variable to capture change in the business environment.

6. Developing the Regression and interpreting the Effect Size First, you calculate the regressed Y estimates to construct its historical trend. Second, you calculate a second set of Y estimates but without the “treatment” period dummy variables. Third, you observe the Effect Size as the difference between Actuals minus Y estimates without the dummy variables.

7. Building the model I am measuring a variable Y that trended upward from 80 in January 2007 to 95 in August 2009. The most recent three months, to the right of the red dotted line, consist of the “treatment” period. The latter had an effect. And, its “Effect Size” is growing as the store is performing above trend.

8. Quantifying the Effect Size The Effect Size of the treatment for the first month (June 2009) is equal to the regressed coefficient 0.74, as the treatment bumped the Y value by that much. Using the standard error of the coefficient, the output builds a 95% confidence interval (0.42 to 1.05). The p value of this coefficient is 0%, so we are nearly 100% confident that this coefficient is different from zero.

9. Advantages of this method It solves the three challenges expressed on slide 2. It generates Effect Size and confidence interval info. It calculates p values similar to the ones generated in a regular hypothesis test. It conducts a test for each month of the “treatment” period reducing the probability that a single test result is erroneous.

10. Technical Appendix

11. Autocorrelation of residuals Time series often break a linear regression assumption that residuals are independent when they are not. Large positive errors follow large positive ones. And, large negative errors follow large negative ones. Autocorrelation of residuals does not affect the regression coefficients. But, it affects the standard error and confidence intervals around such coefficients. To fix this problem use the lagged (t-1) residual of the original regression as an extra variable in a second regression. This example does not exhibit autocorrelation of residuals since the Durbin Watson score is close to 2.0. The seasonality variable and the focus on dummy variables reduce the risk of running into this problem.

12. Heteroskedasticity Linear regression assumes the variance of the errors remains constant (homoskedasticity). Time series often exhibit a change in the variance of errors (heteroskedasticity). Just like autocorrelation of residuals, heteroskedasticity does not affect the regression coefficients. But, it affects the standard error and confidence intervals around the coefficients. To fix this issue, transform the Y variable into a % change or log. This example does not demonstrate heteroskedasticity. The trend variable and the focus on dummy variables reduce the risk of running into this situation. Also, the Y variable being leveled (it increases by only 20% from beginning to end of series) is unlikely to be heteroskedastic.