Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- The Future of Head and Neck Oncolog... by Ninian Peckitt 522 views
- Cochrane Collaboration by Ninian Peckitt 812 views
- Seiten aus KM6_2014_23_12-2 by Ninian Peckitt 296 views
- Randomised Controlled Trials by Ninian Peckitt 1500 views
- Engineering Assisted Surgery - Robo... by Ninian Peckitt 1454 views
- Mini Dental Implants by Ninian Peckitt 1420 views

1,658 views

Published on

Pitfalls in Statistical Analysis - Regression to the Mean

Published in:
Health & Medicine

No Downloads

Total views

1,658

On SlideShare

0

From Embeds

0

Number of Embeds

13

Shares

0

Downloads

28

Comments

0

Likes

1

No embeds

No notes for slide

A regression threat, also known as a "regression artifact" or "regression to the mean" is a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated. The figure shows the regression to the mean phenomenon. The top part of the figure shows the pretest distribution for a population. Pretest scores are "normally" distributed, the frequency distribution looks like a "bell-shaped" curve. Assume that the sample for your study was selected exclusively from the low pretest scorers. You can see on the top part of the figure where their pretest mean is -- clearly, it is considerably below the population average. What would we predict the posttest to look like? First, let's assume that your program or treatment doesn't work at all (the "null" case). Our naive assumption would be that our sample would score just as badly on the posttest as they did on the pretest. But they don't! The bottom of the figure shows where the sample's posttest mean would have been without regression and where it actually is. In actuality, the sample's posttest mean wound up closer to the posttest population mean than their pretest mean was to the pretest population mean. In other words, the sample's mean appears to regress toward the mean of the population from pretest to posttest

It is tempting to assume that treatment is ineffective in those women who are losing bone.

However, because of regression to the mean, most patients (> 80%) who lost bone in the first year of treatment went on to gain bone in the second year despite no change in treatment.

We select a low-pretest scoring sample that has a mean of 30.

Assume that we do not give any program or treatment (i.e., the null case) and that the population is not changing over time on the characteristic being measured (i.e., steady-state).

Given this, we would predict that the population mean would be 50 and that the sample would get a posttest score of 30 if there was no regression to the mean.

Now, assume that the correlation is .50 between the pretest and posttest for the population.

Given our formula, we would expect that the sampled group would regress 50% of the distance from the no-regression point to the population mean, or 50% of the way from 30 to 50.

In this case, we would observe a score of 40 for the sampled group, which would constitute a 10-point pseudo-effect or regression artefact

Regression to the mean is one of the trickiest threats to validity. It is subtle in its effects, and even excellent researchers sometimes fail to catch a potential regression artefact.

Further Reading Computer Simulations for Research Design

http://www.socialresearchmethods.net/simul/simul.htm

http://www.socialresearchmethods.net/simul/reg_m.htm

http://www.socialresearchmethods.net/simul/reg_c.htm

Faculty of Dental Surgery, Royal College of Surgeon’s guidelines

Inappropriate Procedures ≈ 22%

England 1994/95

36,000 inpatient

60,000 day case admissions

Wales (1998/99) for Wales

3000 procedures

Total cost to the NHS in England and Wales ≈ £12 million / year.

van der Linden et al 80 1995

Review of 1001 patients

8’s removed aged 13-75 years

Caries in 7.1% impacted 8’s 69

Caries in 42.7% of adjacent molars 68

- 1. Healthcare Management Regression to the Mean Ninian Peckitt FRCS FFD RCS FDS RCS FACCS Oral and Maxillofacial Surgeon / Facial Plastic Surgeon Adjunct Associate Professor of Engineering Assisted Surgery Massey University New Zealand
- 2. Regression to the Mean His pain got worse, he went to a doctor, and the pain subsided a little. Therefore, he benefited from the doctor's treatment. The frequency of accidents on a road fell after a speed camera was installed. Therefore, the speed camera has improved road safety.
- 3. Regression to the Mean Regression toward the Mean • depends on the way the term is mathematically defined • if a variable – is extreme on its first measurement – it will tend to be closer to the average on a second measurement • Paradoxical – if it is extreme on a second measurement – will tend to have been closer to the average on the first measurement
- 4. Regression to the Mean To avoid making wrong inferences, • regression toward the mean in: – designing experiments – interpreting experimental survey, – and other empirical data (physical, life, behavioral and social sciences) • Regression to the Mean is mathematically defined.
- 5. Regression to the Mean Regression toward the Mean …..has also been called – reversion to the mean and – reversion to mediocrity
- 6. See speaker notes for legend http://www.socialresearchmethods.net/kb/regrmean.php
- 7. 100-Item True False variables • Suppose that all choose randomly on all questions. • Then, each score would be a realization of – one of a set of independent and identically distributed random variables, – with a mean of 50. • Some score >50 Some score <50 just by chance • Top scoring 10% and do second test – on which they again choose randomly – mean score would again = 50.
- 8. 100-Item True False variables • Thus the mean “regresses” all the way back to the mean of the original test. – No matter what scores on the original test – the best prediction of the score on the second test is 50 • If there were no luck or random guessing involved with the test questions – Scoring is the same on the second test as the original test – there would be no regression toward the mean
- 9. Realistic Tests • Most realistic situations fall between these two extremes: • Test scores as a combination of skill and luck • Subset scoring above average would be composed of – those who were skilled and had not especially bad luck – together with unskilled, but were extremely lucky On a retest of this subset – the unskilled unlikely to repeat their luck – skilled have a second chance to have bad luck • Hence, those who did well previously are unlikely to do quite as well in the second test.
- 10. Example – Regression to the Mean • Students two editions of the same test on two successive days. • worst performers on the first day will tend to improve their scores on the second day • best performers on the first day will tend to do worse on the second day. • Scores are determined in part by – underlying ability – in part by chance • For the first test – some will be lucky, and score more than their ability – some will be unlucky and score less than their ability – Some lucky scores will be lucky again on the second test – but more scores will be average or below average – Therefore lucky score (first test) - more likely to have a worse score on the second test – scores less than the mean on the first test will tend to increase on the second test
- 11. Francis Galton Regression towards mediocrity in hereditary stature • extreme characteristics (e.g., height) in parents - not passed on completely to their offspring • Characteristics in the offspring regress towards a mediocre point (Mean) • Galton was able to quantify regression to the mean, and estimate the size of the effect. • “the average regression of the offspring is a constant fraction of their respective mid-parental deviations”. • Child / Parents variable difference proportional to its parents' deviation from typical people in the population. • Parents are each two inches taller than the averages – Child will be shorter than its parents by some factor – regression coefficient) times two inches – Galton estimated this coefficient to be about 2/3: – height will measure around a mid-point that is two thirds of the parents’ deviation from the population average.
- 12. Regression to the Mean Regression towards the Mean • in all bivariate normal distributions • any random variation e.g. height of child vs parents • if the correlation ≠ 1 • predictions must regress to the mean • regardless of the underlying mechanisms • eg. inheritance, race or culture.
- 13. Importance • Regression toward the mean is a significant consideration in the design of experiments. • 1,000 individuals of a similar age scored on risk of a heart attack. • Statistics could be used to measure the success of an intervention on the 50 who were rated at the greatest risk. • The intervention could be a change in diet, exercise, or a drug treatment • Even if the interventions are worthless • the test group would be expected to show an improvement on their next physical exam, – because of regression toward the mean • The best way to combat this effect is to divide the group randomly into a treatment group that receives the treatment, and a control group that does not. • Treatment effective only if the treatment group improves more than the control group.
- 14. Misunderstandings Regression toward the Mean - misused • score >70 on both tests to pass – Score>70 - strong incentive to study / concentrate • Score<70 the first time would have no incentive to do well – might score worse on average the second time. • Possible movement away from 70 – scores <70 it getting lower – scores >70 getting higher • changes between the measurement times may ….. – Augment / Offset / Reverse ……the statistical tendency to regress toward the mean.
- 15. Caution Errors in Interpretation Education • praise for good work – noticed to do more poorly on the next measure • Punished for poor work – were noticed to do better on the next measure. The educators decided to • stop praising • keep punishing on this basis Error of Concept • Regression toward the Mean is not based on cause and effect • Based on random error in a natural distribution around a mean
- 16. Caution – Errors in Interpretation • Although individual measurements regress toward the mean • the second sample - no closer to the mean than the first • Suppose their tendency is to regress 10% of the way toward the mean of 80, • Score of 100 the first day is expected to score 98 the second day • Scored 70 the first day is expected to score 71 the second day • Those expectations are closer to the mean, on average, than the first day scores • But the second day scores will vary around their expectations – some will be higher and some will be lower – This will make the second set of measurements farther from the mean, on average, than their expectations – The effect is the exact reverse of regression toward the mean, and exactly offsets it.
- 17. Caution Errors in Interpretation So for every individual • expect 2nd score closer to the mean than 1st score But for all individuals • expect average distance from the mean to be the same on both sets of measurements Regression toward the mean works equally well in both directions • Highest test score 2nd Day - worse 1st Day. • Best score 1st Day vs Best Score 2nd Day • tendency to regress toward the mean going in either direction • Expect the best scores on both days to be equally far from the mean.
- 18. Wrong Conclusions Many phenomena attributed to the wrong causes when regression to the mean is not taken into account The Triumph of Mediocrity in Business, Horace Secrist 1933 • mountains of data to prove that: • Profit Rates of Competitive Businesses tend toward the Average over Time No such effect: • the variability of profit rates is almost constant over time • Description is the Common Regression toward the Mean
- 19. Wrong Conclusions • Improvement Scores - Standardized Educational Tests in Massachusetts 1999-2000 • Schools were given improvement goals • Difference in the average score achieved by students • Most of the worst-performing schools had met their goals • Department of Education took as confirmation of the soundness of their policies • Many “Best Schools” (e.g Brookline High - 18 National Merit Scholarship Finalists) – were declared to have failed • Findings - Case of Regression to the Mean
- 20. Wrong Conclusions • The psychologist Daniel Kahneman, winner of the 2002 Nobel prize in economics, pointed out that regression to the mean might explain why rebukes can seem to improve performance, while praise seems to backfire.[7] “ I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.” This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.
- 21. Regression to the Mean - Healthcare • Regression to the Mean • widespread statistical phenomenon • potentially serious implications for Healthcare. • May result in wrong conclusions e.g. effect is due to treatment when it is due to chance. • Ignorance will lead to errors in decision making http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 22. Regression to the Mean - Healthcare • The less correlated the two variables….. ……the larger the effect of regression to the Mean • The more extreme the value from the population mean… …..the more room there is to Regress to the Mean • Regression to the Mean occurs when…. – a group is selected with extreme values for one variable – and another variable is then measured. 1,2 • All Healthcare Professionals need to be aware of Regression to the Mean – as it has wide ranging effects http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 23. Regression to the Mean - Healthcare • Clinicians use diagnostic tests to target and monitor treatment. • Regression to the Mean can confound this strategy. • The preliminary test has a high probability of giving an abnormal result through chance • Initial Treatment may be unnecessary • Chance effect – high probability that subsequent measurements will spontaneously regress towards the mean value • This misleads clinicians and patients into thinking that treatment has been effective when – the treatment was either not required – The treatment was ineffective http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 24. Regression to the Mean - Healthcare • Regression to the mean affects all aspects of Health Care • Any intervention aimed at a group or characteristic that is very different from the average will appear to be successful because of Regression to the Mean • In clinical practice, the phenomenon can lead to – misinterpretation of results of tests – new treatments – the placebo effect • Public health interventions are often aimed at sudden increases in disease and thus vulnerable to the effects of Regression to the Mean http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 25. Example See Speaker Notes Percentage changes in bone mineral density among women treated with alendronate or raloxifene after 12 and 24 months, showing regression to the mean at 24 months Women are grouped according to the percentage change at 12 months, and values for 24 months are percentage change from value at 12 months http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 26. New Treatments • Rx New treatments on Patients who are most ill. – Usually produce a gratifying and sharp response – Because of Regression to the Mean • Clinical Trials : – excluded patients who were resistant to treatment – treatments outside the licensed conditions are used – mistaken impression that the new treatment is even better
- 27. Placebo Effect • Trials of hormone replacement therapy show a strong placebo effect on menopausal symptoms.6 • This implies that menopausal symptoms are susceptible to placebo treatment. • Systematic review of placebo vs “open” no treatment - found little evidence for the placebo effect7 • Placebo effect is simply regression to the mean. • Patients in trials of hormone replacement therapy typically score highly on a symptom index • Identification of women with relatively extreme menopausal symptoms: – once treatment starts, improvement will occur in both the placebo and active treatment groups – because of regression to the mean – as patients with the worst clinical scores have the biggest placebo effect http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/
- 28. Public Health • Public health interventions • driven by unexpected increases in incidence of disease • eg response to a sudden rise in traffic incidents • Because a sudden peak in road crashes is often due to chance • changes in policy, such as more rigorous enforcement of speed laws – will reduce crashes because of regression to the mean • The policy of vaccinating children against meningitis - introduced at a time of heightened incidence (see bmj.com) – The Benefit of a 75%-90% reduction in cases 8 is an overestimate – Most of the reduction - due to regression to the mean
- 29. Meningitis Quarterly figures for laboratory confirmed cases of meningitis C in England and Wales • Mass vaccination occurred at a time of increased incidence • So the benefits of immunisation seemed greater
- 30. Decision Making in Healthcare “If public health physicians wish to prove their worth, our advice is that they focus their efforts on a group of problems that are much worse than the national average or have shown an unexpected increase as there will usually be an improvement. It is important, however, to focus on a group of outliers to guarantee an effect because there is a chance that regression to the mean will not affect the results of a single outlier as it is a group phenomenon”. Effect of regression to the mean on decision making in health care Veronica Morton, research fellow1 and David J Torgerson, director1 BMJ. 2003 May 17; 326(7398): 1083–1084.
- 31. Healthcare Management • Regression to the mean can justify league table initiatives for improving poorly performing hospitals. • When poor hospitals are helped by allocating them more resources, regression to the mean will ensure that most will suddenly climb the league table. • In contrast, hospitals at the top of the league table who are rewarded with increased resources for their efforts will fall in the table. • If governments want to justify any initiative, it is better to target those at the bottom of the league than those at the top. • For the individual hospital manager, the problem is more complex. • Those who manage the worst hospitals are likely to see an improvement and thus enhance their careers • However, because regression to the mean is a group phenomenon, the improvement is not certain, and some hospitals will move in the opposite direction.
- 32. Clinical Audit • An audit might identify patients that were operated on by a particular surgical team and had unexpectedly poor results, such as increased postoperative infections. • Implementation of a policy of aggressive procedures to control infection will again often seem to work because of regression to the mean
- 33. Formula Percent of Regression to the Mean Formula Estimation of the percent of regression to the mean in any given situation Prm = 100(1 – r) Prm = the percent of regression to the mean r = the correlation between the two measures
- 34. Formula Consider the following four cases: if r = 1, there is 0% regression to the mean – the two variables are perfectly correlated – and there is no regression to the mean.
- 35. Formula Consider the following four cases: if r = 0.5, there is 50% regression to the mean the sampled group moves 50% of the distance from the no-regression point to the mean of the population
- 36. Formula Consider the following four cases: if r = 0.2, there is 80% regression to the mean – If the correlation is a small 0.20 – the sample will regress 80% of the distance
- 37. Formula Consider the following four cases: if r = 0, there is 100% regression to the mean – if there is no correlation between the measures – the sample will "regress" all the way back to the population mean
- 38. Formula • if r = 0, there is 100% regression to the mean i.e. zero correlation • knowing a score on one measure • gives no information of the likely score on a 2nd measure • Best guess for performance on 2nd measure….. • …..will be the mean of that 2nd measure
- 39. Calculation requirements: • the mean of the sample on the first measure • the population mean on both measures • the correlation between measures See Speaker Notes Estmation and Correction of Regression to the Mean
- 40. Estmation and Correction of Regression to the Mean Assume gain of 15 points (average) Gain across entire distribution If r=1 (no Regression to Mean) • Post Test 30+15 = 45 Correlation .5 = 50% Regression • Post Test 50% of Means 45 – 65 • Post Test Average of 55 • Pseudo effect of 10 points See Lecutre Notes
- 41. Solutions ? • Understanding the phenomenon • Whenever possible, Policy should be based on evidence from trials • The effectiveness of management league tables could be tested by randomising poorly performing hospitals to new management or extra resources • This would tell us which intervention was most effective • In clinical practice, sequential testing to get an average value, which most doctors would do for blood pressure, is a solution for some tests.
- 42. NICE NICE Guidelines for Wisdom Teeth Removal • Political Motivation • Postive Bias • Financially Motivated • Premature Conclusions • RCTs have not been undertaken ! • RCT Data will take 10 years to collect ! See Speaker Notes
- 43. 0% 10% 20% 30% 40% 50% 60% Pain Pericoronitis Caries 8 Caries 7 P/A Path Perio Root Resorbtion Cyst Tumour 53% 59% 7% 42% 5% 5% 5% 11% 2% Wisdom Teeth Removal - Reasons Pain Pericoronitis Caries 8 Caries 7 P/A Path Perio Root Resorbtion Cyst Tumour
- 44. Wisdom Teeth - NICE Increased Risk of Caries • Van der Linden 1995 (Removal 8’s) • Caries 8’s (7.1%) • Caries Adj Molars (42.7%) • Risk with 8/7 Contact • “Best Practice vs Negligence?
- 45. Regression to the Mean • It is a statistical phenomenon • It is a group phenomenon • It happens between any two variables • It is a relative phenomenon • You can have regression up or down • The more extreme the sample group, the greater the regression to the mean • The less correlated the two variables, the greater the regression to the mean.
- 46. Thank you for your Attention

No public clipboards found for this slide

Be the first to comment