Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Automated stress tests for econometric models Presentation to Stata 2008 Fall North American Users Group Meeting by Roy J. Epstein, PhD Adjunct Professor of Finance, Boston College November 14, 2008
  2. 2. Need for Greater Quality Control and Transparency in Applied Econometrics <ul><li>Econometric models now part of mainstream in policy analysis, legal proceedings, and other advocacy situations </li></ul><ul><li>But assessing reliability of estimation results often difficult </li></ul><ul><ul><li>Lack of transparency  Risk of bad decisions </li></ul></ul><ul><li>Enhanced regression software—a virtual referee—is key to improving the practice of econometrics </li></ul>
  3. 3. Regression Software as Virtual Referee for Econometric “Best Practices” <ul><li>Journal referee process and professional dialogues routinely probe effects of different: </li></ul><ul><ul><li>Explanatory variables </li></ul></ul><ul><ul><li>Data </li></ul></ul><ul><ul><li>Statistical procedures </li></ul></ul><ul><li>Similar quality control process can/should be built into regression software </li></ul><ul><ul><li>Promotes neutral and transparent assessment of reliability </li></ul></ul>
  4. 4. Automating Regression Stress Tests <ul><li>Software can anticipate reasonable perturbations of a model </li></ul><ul><li>Program then can assess sensitivity of original regression to the perturbations </li></ul><ul><li>Can be standard part of regression output </li></ul><ul><ul><li>Much as we now expect to see t-statistics </li></ul></ul>
  5. 5. Extending Stata for Stress Tests <ul><li>Macro facility allows design of highly flexible testing strategy </li></ul><ul><li>Can implement as stand-alone post-estimation routine </li></ul><ul><li>Probably more advantageous to incorporate as part of standard output </li></ul><ul><li>Can reflect ongoing professional discussion of “best practice” stress tests in econometrics </li></ul>
  6. 6. A Taxonomy of Stress Tests <ul><li>Algebraic sign of regression coefficient </li></ul><ul><li>Magnitude </li></ul><ul><li>Statistical significance </li></ul><ul><li>Panel models: </li></ul><ul><ul><li>Validity of pooling assumptions </li></ul></ul><ul><ul><li>Cross-sectional error correlation </li></ul></ul><ul><li>Autocorrelation </li></ul><ul><li>Temporal aggregation </li></ul><ul><li>Lagged variables </li></ul><ul><li>Dropping variables </li></ul><ul><li>Outliers </li></ul><ul><li>Functional form </li></ul><ul><li>Weighting </li></ul>
  7. 7. Stata Command (Prototype) <ul><li>stresstest price focuscoef cost realgdp i.firm i.density i.month, timeseries(year month) panel(firm density) w(pounds); </li></ul><ul><li>Based on “regress” command syntax </li></ul><ul><li>Parameters identify time, panel, and weighting variables as appropriate </li></ul><ul><li>Implemented as conventional Stata 10 ado file </li></ul>
  8. 8. Case Studies
  9. 9. Conwood v. US Tobacco <ul><li>Econometric analysis of tobacco sales </li></ul><ul><ul><li>Logic of plaintiff model was such that the more negative the estimated slope, the greater the damages. </li></ul></ul><ul><li>$1 billion claimed damages, after trebling </li></ul><ul><li>Sustained after review by Supreme Court </li></ul><ul><li>Timely use of stress tests probably would have vindicated defendant </li></ul>
  10. 10. The Conwood Data and Regression Line Washington, DC
  11. 11. Stress Test Easily Identifies Effect of Outlier Negative estimated slope, $1 billion damages Positive estimated slope, (excludes Washington, DC) Zero damages
  12. 12. Confidential Antitrust Case <ul><li>Alleged price-fixing conspiracy </li></ul><ul><li>Panel model of average prices (by firm, product, month) </li></ul><ul><li>Nearly 100 control variables </li></ul><ul><li>Claimed damages of $330 million based on “statistically significant” coefficient of dummy variable for conspiracy period </li></ul>
  13. 13. Why the Virtual Referee Would Have Helped <ul><li>Plaintiff expert: </li></ul><ul><ul><li>Did not test validity of pooling data for 4 different firms </li></ul></ul><ul><ul><li>Did not report results from aggregated model but admitted having estimated it </li></ul></ul><ul><ul><li>Ignored serial correlation in the residuals </li></ul></ul><ul><ul><li>Did not report effect of excluding control variables with dubious justification </li></ul></ul><ul><li>Stress tests quickly cut through complexity of model to identify serious reliability problems </li></ul>
  14. 14. Stress Tests Reveal High Sensitivity to Pooling Data from Different Firms
  15. 15. Stress Tests Reveal High Sensitivity to Aggregating Across the Panel
  16. 16. Stress Tests Reveal High Sensitivity to Autocorrelation
  17. 17. Stress Tests Reveal High Sensitivity to Presence of Certain Variables
  18. 18. Summary <ul><li>Software designers have important role in improving the practice of econometrics </li></ul><ul><li>Regression results not reliable until sensitivities identified and explained </li></ul><ul><li>Software can and should automatically invoke appropriate best practice stress tests </li></ul><ul><li>Software will act as virtual referee </li></ul>
  19. 19. For Further Information… <ul><li>Roy J. Epstein, PhD </li></ul><ul><li>Expert economic analysis for complex litigation </li></ul><ul><li>1280 Massachusetts Ave., 2nd Fl. </li></ul><ul><li>Cambridge, MA 02138 </li></ul><ul><li>[email_address] </li></ul><ul><li>(617) 489-3818 </li></ul><ul><li>Adjunct Professor of Finance, Boston College </li></ul>