More Related Content
What's hot
What's hot (20)
Similar to 09 selection bias
Similar to 09 selection bias (20)
More from Abdiwali Abdullahi Abdiwali
More from Abdiwali Abdullahi Abdiwali (20)
Recently uploaded
Recently uploaded (20)
09 selection bias
- 1. 3/15/2011 Sources of error: Selection bias 1 Sources of error: Selection bias Victor J. Schoenbach, PhD Department of Epidemiology Gillings School of Global Public Health University of North Carolina at Chapel Hill www.unc.edu/epid600/ Principles of Epidemiology for Public Health (EPID600)
- 2. 2 Excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college Richard Lederer, St. Paul's School http://www.edfac.usyd.edu.au/staff/souters/Humour/Student.Blooper.World.html
- 3. 3 Excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college Richard Lederer, St. Paul's School “Finally Magna Carta provided that no man should be hanged twice for the same offense.” “Another story was William Tell, who shot an arrow through an apple while standing on his son's head.”
- 4. 4 Excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college Richard Lederer, St. Paul's School “It was an age of great inventions and discoveries. Gutenberg invented removable type and the Bible. Another important invention was the circulation of blood. Sir Walter Raleigh is a historical figure because he invented cigarettes and started smoking. And Sir Francis Drake circumcised the world with a 100 foot clipper.”
- 5. 5 Winning tongue in cheek headlines selected for humorous magazine cover contest (Epimonitor.net) •“Don’t Ask, Don’t Tell---Double Blinding In Military Studies” (Melissa Adams) •Sensitivity Analysis---Are You Really A New Man? (Mark Colvin) •Absent Sex Life? Lucky You! 38 Sexually Transmitted Diseases You Won’t Get (Mark Colvin) •Do You Have Survey Phobia? Take Our Quiz and Find Out!! (Mary Anne Pietrusiak)
- 6. 3/15/2011 Sources of error: Selection bias 6 “To error is human” • Science emphasizes systematic, repeatable, carefully-conducted observation • Laboratory investigations are highly controlled, to minimize unwanted influences • Human sciences must contend with many threats to validity www.cfa.harvard.edu/poem/
- 7. 10/29/2001 Sources of error: Selection bias 7 “To error is human” Any epidemiologic study presents many, many opportunities for error in relation to: • Selection of study participants • Classification and measurement • Comparison and interpretation
- 8. 3/15/2011 Sources of error: Selection bias 8 Accuracy, Bias, Error, Precision, Reliability, Validity • Accuracy: lack of error – getting the correct result • Bias: systematic error – independent of study size • Error: discrepancy between the observed result and the true value • Precision: absence of random error • Reliability: repeatability of a measure • Validity: absence of bias (or absence of all error)
- 9. 10/29/2001 Sources of error: Selection bias 9 Internal validity versus external validity • Internal validity: whether the study provides an unbiased estimate of what it claims to estimate • External validity: whether the results from the study can be generalized to some other population
- 10. 10/18/2004 Sources of error: Selection bias 10 “Random error” • “that part of our experience that we cannot predict” (Rothman and Greenland, p. 78) • Usually most easily conceptualized as sampling variability
- 11. 10/29/2001 Sources of error: Selection bias 11 Draw 1,000 random samples from population with 50% women Estimate of % Women Distribution N = 100 N = 1,000 Highest 68 54.9 10% above 56 above 51.9 25% above 53 above 51.0 25% below 47 below 48.9 10% below 44 below 48.0 Lowest 33 45.0
- 12. 7/30/2010 Sources of error: Selection bias 12 Random error can be problematic, but . . . • Influence can be reduced – increase sample size – change design – improve instrumentation • Probability of some types of influence can be quantified (e.g., confidence interval width)
- 13. 10/29/2001 Sources of error: Selection bias 13 Systematic error (“bias”) is more problematic • May be present without investigator being aware • Sources may be difficult to identify • Influence may be difficult to assess
- 14. 10/19/2010 Sources of error: Selection bias 14 Direction of bias - “which way is up?” • Positive bias – observed value is higher than the true value • Negative bias – observed value is lower than the true value • Bias towards the null – observed value is closer to 1.0 than is the true value • Bias away from the null – observed value is farther from 1.0 than is the true value True Observed TrueObserved Observed TrueNull True Observed1 NullObserved2
- 15. 10/29/2001 Sources of error: Selection bias 15 Classifying types of bias • Selection bias – differential access to the study population • Information bias – inaccuracy in measurement or classification • Confounding bias – unfair comparison
- 16. 10/18/2004 Sources of error: Selection bias 16 Possible sources of selection bias • Participant selection procedure, e.g., exposure affects case ascertainment (“detection bias”) or control selection • Differential unavailability due to death (“selective survival”), illness, migration, or refusal (nonresponse bias) • Loss to follow-up / attrition / missing data
- 17. 10/16/2007 Sources of error: Selection bias 17 Case-control studies • Case-control studies are highly vulnerable to selection bias, particularly in the control group. • The purpose of the control group is to estimate exposure in the base population. • Selection bias results if control selection is not neutral with respect to exposure.
- 18. 10/29/2001 Sources of error: Selection bias 18 Example: differential loss to follow-up (cohort study) Complete cohort Observed cohort Cases Noncases Cases Noncases Exposed 40 160 32 144 Unexposed 20 180 18 162 Total 60 340 50 306 Risk ratio 2.0 1.8
- 19. 7/7/2002 Sources of error: Selection bias 19 Example: non-response bias in case-control study Target population Study population Cases Pop-time Cases Noncases Exposed 200 20,000 180 200 Unexposed 400 40,000 300 400 Total 600 60,000 480 600 Odds ratio 1.0 1.2
- 20. 7/1/2009 Sources of error: Selection bias 20 Examples • Case-control study of socioeconomic factors and risk of anencephaly in Mexico • Case-control study of physical fitness and heart attacks • Cohort study of smoking cessation methods (Free & Clear)
- 21. 3/15/2011 Sources of error: Selection bias 21 Selective survival can affect a cohort study Natural history of HIV infection 1 Recruit participants at the time they become HIV infected 2 Recruit participants by means of a serologic survey to detect prevalent infections. People who progress to AIDS and die relatively soon after infection will be underrepresented in study 2.
- 22. 10/25/2005 Sources of error: Selection bias 22 Prevalent cohort – cohort of survivors Effect of hypertension in an elderly cohort If people who die before becoming elderly were more vulnerable to end-organ damage from hypertension, then the cohort study may observe less morbidity associated with hypertension than would be observed if the study had enrolled younger participants.
- 23. 3/15/2011 Sources of error: Selection bias 23 Example – cohort of survivors Effect of environmental tobacco smoke (ETS) in newborns If ETS increases early fetal losses, possibly undetected, then the most susceptible fetuses may be unavailable to be part of the cohort study of effects of ETS on infants. In this example, the early fetal losses are “informative”.
- 24. 10/29/2001 Sources of error: Selection bias 24 Conceptual framework (Kleinbaum, Kupper, Morgenstern) • Target population • Actual population • Study population • “Selection probabilities”
- 25. 10/29/2001 Sources of error: Selection bias 25 Target population vs. actual population • Target population – the population from which we think we are studying a sample • Actual population – the population from which we are actually studying a sample • Study population – a random sample from the actual population
- 26. 10/18/2004 Sources of error: Selection bias 26 Actual population vs. study population • “Selection probabilities” – the probability that someone in the target population will be in the actual population Selection probability (for each exposure-disease subgroup) = probability (target actual)
- 27. 10/29/2001 Sources of error: Selection bias 27 Selection probabilities from the target population to the actual population Target population Actual population Exposed Unexposed Exposed Unexposed Cases A B A o B o Noncases C D C o D o alpha = A o /A, beta = B o /B, gamma = C o /C, delta = D o /D
- 28. 10/29/2001 Sources of error: Selection bias 28 Example of selection probabilities Endometrial cancer and estrogen use • Case-control studies found high OR’s • Horwitz and Feinstein claimed “detection bias”: women may have asymptomatic tumors; if they have vaginal bleeding from estrogens their cancer will be discovered
- 29. 6/27/2002 Sources of error: Selection bias 29 Horwitz and Feinstein’s argument Target population Actual population Estrogen Unexposed Estrogen Unexposed Endometrial cancer 2,000 8,000 1,800 900 Noncases 2,000,000 8,000,000 1,600,000 6,400,000 OR 1.0 8.0
- 30. 6/23/2002 Sources of error: Selection bias 30 Horwitz and Feinstein’s argument Actual population / Target population Estrogen Unexposed Cases alpha = 1,800 / 2,000 beta = 900 / 8,000 Noncases gamma = 1,600,000 / 2,000,000 delta = 6,400,000 / 8,000,000
- 31. 10/18/2004 Sources of error: Selection bias 31 Horwitz and Feinstein’s argument Actual population / Target population Estrogen Unexposed Cases alpha = 0.9 beta = 0.11 Noncases gamma = 0.8 delta = 0.8 alpha/beta = 8 > gamma/delta = 1
- 32. 6/27/2002 Sources of error: Selection bias 32 Horwitz and Feinstein’s solution Actual population / Target population Estrogen Unexposed Cases alpha = 0.9 beta = 0.11 Noncases gamma = 0.36 delta = 0.06 alpha/beta = 8, gamma/delta = 6
- 33. 10/24/2006 Sources of error: Selection bias 33 Summary • Epidemiologists differentiate between random error and systematic error (“bias”). • Bias: selection, information, and confounding. • Bias can arise in any type of epidemiologic study. • Ask: bias from what, how much, what effect?
- 34. 34 Excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college Richard Lederer, St. Paul's School “The greatest writer of the Renaissance was William Shakespeare. He was born in the year 1564, on his birthday. He never made much money and is famous only because of his plays. He wrote tragedies, comedies, and hysterectomies, all in Islamic pentameter. Romeo and Juliet are an example of a heroic couplet. Romeo's last wish was to be laid by Juliet.”
- 35. 35 Excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college Richard Lederer, St. Paul's School “Writing at the same time as Shakespeare was Miguel Cervantes. He rote Donkey Hote. The next great author was John Milton. Milton wrote Paradise Lost. Then his wife died and he wrote Paradise Regained.”
Editor's Notes
- Greetings. Namaste, konnichiwa, drasvuitsia, Ni-hau, merhaba, oseo This lecture is on Sources of Error, particularly Selection Bias.
- To put us in the right frame of mind, here are some excerpts from (allegedly) actual student history essays collected by teachers from 8th grade through college. The come from a web page by Richard Lederer, St. Paul’s School, www.edfac.usyd.edu.au/staff/souters/Humour/Student.Blooper.World.html
- “Finally Magna Carta provided that no man should be hanged twice for the same offense.” “Another story was William Tell, who shot an arrow through an apple while standing on his son's head.”
- “It was an age of great inventions and discoveries. Gutenberg invented removable type and the Bible. Another important invention was the circulation of blood. Sir Walter Raleigh is a historical figure because he invented cigarettes and started smoking. And Sir Francis Drake circumcised the world with a 100 foot clipper.”
- Here are some winning tongue in cheek headlines selected for humorous magazine cover contest: “Don’t Ask, Don’t Tell---Double Blinding In Military Studies” (Melissa Adams) Sensitivity Analysis---Are You Really A New Man? (Mark Colvin) Absent Sex Life? Lucky You! 38 Sexually Transmitted Diseases You Won’t Get (Mark Colvin) Do You Have Survey Phobia? Take Our Quiz and Find Out!! (Mary Anne Pietrusiak) cover contest (Epimonitor.net)
- Science emphasizes systematic, repeatable, carefully-conducted observation. By designing and performing experiments, a scientist attempts to control all aspects of a situation in order to observe the effect of varying only those elements s/he is trying to investigate. My brother-in-law is an astrophysicist at the Harvard-Smithsonian Center for Astrophysics in Cambridge, Massachusetts. He has been conducting an experiment to use laser interferometry to verify the law of falling objects to a higher degree of precision than has heretofore has been obtainable. A great deal of the experimental set-up consisted in insulating highly-sensitive measurement apparatus from outside perturbations such as vibration of the building. Measurements had to be made at night, since the apparatus would pick up even the gravitational force from a car parking outside the building. Epidemiology and other human sciences have nowhere near the precision of measurement that is available in the physical sciences and nowhere near the ability to control threats to validity. So recognizing and minimizing the effect of myriad sources of error is a constant theme in epidemiologic research.
- Sources of systematic error in epidemiologic studies are generally considered under three headings: errors that may arise from the way that study participants are selected into the study, errors from classification and measurement procedures, and errors from comparisons and their interpretation. These sources of systematic error are in addition to sampling variability, which is often termed “random error”.
- The more important the concept, the richer the vocabulary that evolves to express its various aspects. Given the importance of error, we would expect a wealth of terms to characterize it, and we will not be disappointed in that expectation. Here are some of the major terms, including several that we have already encountered. Accuracy is a general term for getting the right result. Accuracy is the opposite of error, which is used to refer to a discrepancy between an observed result and the true value. Of course, the true value is rarely known. Error is typically regarded as arising from two types of processes – random processes, such as simple random sampling or flipping a fair coin, and systematic processes which move an estimate in one direction. Epidemiologists use the term bias to refer to systematic error. Random error can be reduced by increasing sample size. As the sample size increases, observed results tend to converge toward their underlying values. Systematic error, on the other hand, is unaffected by sample size. Precision refers to the absence of random error. Similarly, reliability refers to the extent to which a measurement or finding will be observed consistently by different observers, using different instruments, at different times, or across other circumstances where the underlying construct is believed not to have changed. Validity refers to the absence of systematic error (“bias”) or, sometimes, the absence of all error.
- Epidemiologists sometimes use the terms “internal validity” and “external validity”. Internal validity concerns whether the study provides an unbiased (valid) estimate of what it claims to estimate. For example, suppose investigators say that they are studying Alzheimer’s disease in North Carolina elderly. If the study findings do in fact accurately depict the situation among NC elderly, then the study is (internally) valid. External validity concerns whether the results from a study apply to some wider population that the study may have relevance for but does not claim to estimate. For example, do the results from a study carried out in North Carolina apply to people in the southeast U.S. as a whole, the entire U.S., all of North America, all industrialized countries, etc.? The term generalizability is often used interchangeably with “external validity”.
- The term “random error” is widely employed yet less widely understood. Rothman and Greenland (Modern epidemiology, p78) define it as “that part of our experience that we cannot predict” (some might say the part that we do not attempt to predict). The concept can easily be demonstrated in the form of sampling variability, where we can readily show that a series of samples from a single population will in general differ from one another and from the population, at least slightly.
- Sampling variability can be reduced by increasing sample size. For example, suppose that we have a large population for which we want to estimate the percentage of females, which happens to be 50% in this population. If we draw 1,000 samples of 1,000 people each – that’s the right-hand column in the table – and compute the percentage of women in each sample, we might observe the percentage to range across the samples from as low as 45% to as high as nearly 55%, with about half of the samples having a percentage between about 49% and 51%. On the other hand, if the samples had only 100 people each, the percentages of women in the samples would vary more widely around the population value of 50%. As shown in the left-hand column, about one-fourth of the estimated percentages from the smaller samples would be below 47% and one-fourth above 53%. So the larger sample size has reduced the amount of random error in our estimates of the population sex ratio. In general, if the magnitude of the error can be reduced by increasing the sample size, then we classify the error as random.
- Random error can certainly be problematic, but it is generally not as problematic as systematic error. For one, we know a great deal about random error and how to reduce it. We can, for one, increase sample size. It may also be possible to reduce random error by changing the design of our sampling plan or using more precise measurement instruments. Moreover, the probability of various degrees of influence from random error can be quantified, so that we can use confidence intervals to express the uncertainty inherent in our estimates due to sampling variability. The precision of an estimate is defined as the reciprocal of the standard error of the estimate, but the width of the confidence interval is often used in epidemiology. The difference between the upper and lower 95% confidence limits for a rate or proportion is approximately four (2 x 1.96) times the standard error of the point estimate. The standard errors for many ratio measures (e.g., , risk ratio, rate ratio, odds ratio) are obtained with a log transformation, so the width of their confidence intervals is quantified as the ratio of the upper to the lower confidence limit (called the confidence interval ratio, or CLR). The natural log of the CLR is 2 x 1.96 times the log of the standard error.
- Whatever the problems with people’s understanding of random error and statistics, systematic error, also called “bias”, is more problematic for epidemiologists. Systematic error may be present without the investigator’s being aware of it, and sources of systematic error may be difficult to identify. Even when we can identify sources of error, their influence may be difficult to assess.
- When we talk about systematic error, we are often interested in its direction. Random error, being random, goes in both directions. A specific source of systematic error, however, usually shifts the observed value in a particular direction. So we need some terminology for giving directions. Most often we are referring to bias in an estimated relative risk (incidence density ratio, cumulative incidence ratio, or odds ratio). For a relative risk, the null value (no association) is 1.0. A positive bias means that whatever is the true relative risk (e.g., 2.3, 1.0, 0.5), the observed relative risk is higher, i.e., in a positive direction (e.g., 3.3 observed vs. 2.3 true; 1.3 vs. 1.0, 0.8 vs 0.5). Negative bias is the inverse of positive bias. For the three example true relative risks just quoted, the following observed relative risks would indicate negative bias: 1.9 observed vs. 2.3 true, 0.8 vs. 1.0, 0.3 vs. 0.5). Some error processes tend to bias the measure of relative risk in a positive or negative direction. In other cases, however, the error process may tend to shift the observed relative risk closer to 1.0 (the null value) or farther from 1.0, regardless of whether the direction is positive or negative. For these situations we have the terms “towards the null” and “away from the null”. If the true relative risks were 2.3, 1.0, and 0.5, then bias towards the null would be exemplified by 1.8 observed vs. 2.3 true and by 0.8 vs. 0.5. A true relative risk of 1.0 is not susceptible to bias towards the null since it is already at the null value. That’s handy to know, since it means that a bias towards the null can never create the completely false impression of an association.
- As noted in an earlier slide, epidemiologists usually conceptualize sources of systematic error as falling into one of three categories, and here are the terms used to refer to the types of error that result. Selection bias refers to systematic error resulting from differential access to the study population for different exposure-disease subgroups. Information bias refers to systematic error resulting from inaccuracy in measurement or classification of study variables, including disease, exposure, and other risk factors. Confounding bias differs somewhat from the other two forms, in that with confounding bias the data themselves may be fully accurate. Confounding bias is a systematic error of interpretation that results from making an unfair comparison, i.e., from a failure to take into account important differences between exposed and unexposed groups. We will take up each of these types of bias. The rest of this lecture will examine selection bias. The following lectures take up information bias and confounding.
- Selection bias can arise from any aspect of the procedures for selecting participants for a study. Several common examples are situations where the exposure affects the ascertainment of cases for a case-control study (a situation sometimes referred to as “detection bias”) or affects the selection of controls. For example, doctors are more likely to suspect the presence of heart disease in a man who is overweight, hypertensive, and/or smokes cigarettes, so a middle-aged person with these characteristics who comes to a doctor complaining of chest pain may be more likely to have a thorough workup for angina pectoris than someone without these heart disease risk factors. Another example is what we might call differential unavailability due to death, illness, migration, or refusal. “Selective survival” refers to the fact that people with different characteristics are more or less likely to survive after contracting a disease. Similarly, characteristics of people who are more likely to refuse to participate in a study are therefore likely to be underrepresented. Attrition of participants from a study can produce bias if the incidence rates in people who drop out differ from those in people who complete the study. Loss to follow-up in a cohort study and participants not included in the data analysis due to missing data items are also potential sources of selection bias.
- Case-control studies are highly vulnerable to selection bias, particularly from the identification and recruitment of the control group. Clear thinking and considerable effort are required to avoid introducing bias. The essential purpose of the control group is to provide an estimate of exposure in the base population, the population from which the cases arise. This understanding represents a shift in emphasis from the traditional view of case-control studies, which emphasized the comparability of controls to cases. If the study includes all cases from a defined geographical area, then a representative sample of residents from that area should provide a valid estimate of exposure. However, such a sample will be difficult and expensive to recruit. Also, unless the response rate is high, the sample may yield a biased estimate due to differential nonresponse. Furthermore, a population-based sample may not provide accurate information on certain exposures, such as medical history. A control group drawn from among hospitalized patients will be easier and less expensive to recruit, have a higher response rate, and provide better information about medical exposures. But if the conditions that bring people to the hospital are related to the exposure under study, then the control group and the overall study will be biased. For example, some early case-control studies of lung cancer and smoking underestimated the association because they drew their controls from hospitalized patients.
- Let’s look at selection bias with two numerical examples. Suppose that we are studying a cohort of 400 people, 200 of whom are exposed and 200 of whom are unexposed. The left-hand table shows the complete cohort. 40 of the 200 exposed people develop the disease, compared to 20 of the unexposed people, so the relative risk is 40/200, which is the cumulative incidence of the disease in the exposed, divided by 20/200, which is the cumulative incidence of the disease in the unexposed, so the relative risk is 2.0. Now suppose that there is loss to follow-up among both exposed and unexposed, including both people who will develop the disease and people who will not. Suppose that the proportion who drop out is 10% in all exposure-disease groups except for some reason 20% drop out among people who are exposed and are going to develop the disease (or who were exposed and have developed the disease but not yet been counted as cases). Under this scenario, the right-hand table shows the cohort that we will observe. The relative risk in the observed cohort is 32/176 divided by 18/180, or 1.8, so that due to selection bias the observed relative risk is 10% less than the true relative risk.
- Here is a numerical example of how selection bias due to non-response might distort the observed odds ratio in a case-control study. Again, the left-hand table represents the true situation in the population, while the right-hand table represents what we find in the case-control study. In this case there are 600 cases of the disease – that’s on the left – but only 90% of exposed cases agree to be in the study, so that in the upper left hand corner, of the 200 exposed cases, only 90%, or 180, agree to be in the study and therefore appear in the right-hand table, the study population, and let’s assume that only 75% of unexposed cases participate, so of the 400 in the lower left of the table only 300, or 75%, agree to come into the study population. Then the odds ratio in the case-control study will be 1.2, suggesting an association where in fact none exists.
- Let’s consider some examples of studies where selection bias would occur. Muñoz et al. conducted a case-control study of the risk of anencephaly (a congenital neural tube defect) and socioeconomic status (SES) in Mexico. Cases were newborns diagnosed with anencephaly (or for stillbirths, with anencephaly listed as the cause of death) and reported to a national neural tube defects registry. Controls were obtained from live newborns in the same hospital systems. Mothers were contacted and interviewed regarding SES. However, low SES mothers were less likely to be interviewed due to less-than-reliable reporting to the national registry in low SES areas. If low SES increased the risk of anencephaly, the cases included in the study would underrepresent cases from low SES families, biasing the estimated association (Muñoz JB, et al. Socioeconomic factors and the risk of anencephaly in a Mexican population: A case-control study. Public Health Reports 2005; 120:39-45.) Or, suppose we are conducting a case-control study of the protection from heart attack among people who are physically fit. Cases are persons newly-admitted to the coronary care unit at area hospitals. Controls are obtained through random digit dialing in that same geographical area. Since physical fitness is protective, we should find the prevalence of being physically fit to be lower among the heart attack victims than in the population control group. However, many people who suffer a heart attack die before they are admitted to the hospital. What if being physically fit improves the chances of surviving a heart attack? Then hospitalized heart attack victims will tend to be more fit than heart attack victims in general, thereby biasing our odds ratio toward finding less of a benefit (and possibly even a risk) from physical fitness. Here is a third example. A number of years ago some colleagues and I conducted a randomized trial of quit smoking interventions. The usual practice in such studies has been to count dropouts as continuing smokers, in order to have a more conservative estimate of the quit rate. However, what if quitters in the control group felt that they did not receive anything from the study so that they were more likely to drop out than quitters in the group that received the quitting intervention. Then counting all dropouts as smokers would exaggerate the apparent effect of the intervention.
- So selection bias can affect cohort and case-control studies, as well as cross-sectional studies. Ordinarily we think of loss to follow-up as being the primary source of selection bias in cohort studies, since the study base is the cohort of people being followed. However, there are situations in which something like selection bias can occur from other causes. For example, here are two study designs for examining prognosis after HIV infection. In the first, participants are recruited when they are newly-infected with HIV. In the second, participants are recruited through a serologic survey that identifies people with prevalent infections. The second study, which is sometimes called a “prevalent cohort”, will underrepresent people who have an especially rapid course to death, since they will have died before being enrolled. Such a process has been suggested as a reason that the Women’s Health Initiative randomized trial of hormone replacement therapy found higher CVD rates in women taking HRT, whereas observational studies (or prevalent cohorts) had found lower CVD death rates in women taking HRT.
- Another hypothetical example of how selective survival can affect a cohort study is a prospective investigation of end-organ damage (e.g., blindness, kidney disorder) among elderly persons with hypertension. If people who die before becoming elderly were more vulnerable to suffering end-organ damage from hypertension, then a cohort study that enrolls people at age 65 years may observe less morbidity associated with hypertension than would be observed among younger participants. Although the different observations reflect a true difference in what is seen with younger and older hypertensives, the results could easily be misinterpreted.
- Yet another hypothetical example of how selection bias might occur in a cohort of survivors is a situation in which environmental tobacco smoke (ETS) causes early fetal losses, possibly undetected. If the most susceptible fetuses did not survive to birth, a study of the effects of ETS among newborns will see less of an effect that has occurred. These examples fall into the category of what is called “competing risks”. Loss of participants from competing risks, dropout, nonresponse, missing data, and other causes is said to be “noninformative” when losses do not affect estimates (e.g., proportional losses in all groups being compared) or “informative” (meaning that the losses cause bias in the association of interest).
- A helpful conceptual framework for thinking about situations where selection bias may operate is the one elaborated by David Kleinbaum, Larry Kupper, and Hal Morgenstern, in their book Epidemiologic methods: theory and quantitative methods. The KKM framework envisions a “target” population, an “actual” population, a “study” population, and “selection probabilities”.
- The target population consists of the people whom we believe we are studying. If, for example, we are trying to study depression among all teenagers in North Carolina, then the target population consists of all North Carolina teenagers. The actual population consists of the people in the target population who might actually contribute data to our study. For example, if we were collecting our data by telephone interviews with a random sample of households listed in the telephone directory, then the actual population would consist of teenagers who live in households with a listed telephone number, whose parents would give permission for the interview, who would participate if invited, and who would be interviewed (e.g., an interviewer and the participant speak the same language). The study population consists of the people whose data we have collected and analyzed. In the KKM framework, the study population is regarded as an unbiased random sample from the actual population. If the sample size were increased to its maximum, then the study population would be the same as the actual population. This framework divides up sources of error into random error (divergence between the actual population and a particular study population) and systematic error (divergence between the actual population and the target population).
- The final component of the KKM conceptual framework is the selection probabilities that describe the relation between the target population and the actual population. For each exposure-disease subgroup, there is a selection probability that describes the proportion of the target population for each subgroup that is available in the actual population.
- In this slide, the left-hand table represents the target population – the population whose disease-exposure relations we seek to measure. The right-hand table represents the actual population – the proportions of the target population that are actually available to our study. The selection probabilities are the proportions of each subgroup in the target population that make it into the actual population. There is a selection probability for each exposure-disease subgroup. So, for example, if we call the selection probability for exposed-cases “alpha”, as KKM do, then: alpha = Ao / A Similarly, the remaining selection probabilities are beta = Bo / B, gamma = Co / C, and delta = Do / D.
- To see how these selection probabilities can be used, consider the following example based on studies of endometrial (uterine) cancer and postmenopausal estrogens and the argument by Ralph Horwitz and Alvan Feinstein that detection bias was responsible for much or all of the observed increase in risk among users of exogenous estrogens. During the 1970’s, several case-control studies found a strong association between risk of endometrial cancer and prior use of estrogen replacement medications, with odds ratios on the order of 6 - 10. Horwitz and Feinstein proposed that these associations were the result of a “detection bias” in which endometrial cancers were more likely to come to medical attention in women who took estrogen than in women who did not. Exogenous estrogens often cause vaginal bleeding, as a side effect. When a postmenopausal woman experiences bleeding, she is likely to see her doctor and undergo a dilation and curretage (D&C) procedure in which the lining of her uterus is removed and examined. Endometrial cancers are found in this way. Thus, among women with asymptomatic endometrial cancer, if women who take estrogens are more likely to bleed and to have a D&C, then their cancers are more likely to be detected. In this way, detection bias could result in a higher prevalence of estrogen use among detected cases of endometrial cancer than would exist among all cases. For the Horwitz and Feinstein argument to explain the observed endometrial cancer-estrogen association, there would need to be a sizable reservoir of undiagnosed endometrial cancer waiting to be detected.
- Here are some hypothetical data that illustrate this argument as it would be presented in the KKM framework. Again, the left-hand table shows a target (true) population that investigators believe they are studying. In this particular target population of just over 10,000,000 postmenopausal women, there are 10,000 postmenopausal women with endometrial cancer. We assume that 20% of the women – overall and those who develop endometrial cancer – have used postmenopausal estrogens. Case-control studies, however, typically rely on the health-care system to identify cases. If women who take estrogens are more likely to experience bleeding and a D&C, then the proportion of endometrial cancers detected among women who take estrogen (1,800 / 2,000 = 90%) would be higher than the proportion among women with endometrial cancer who do not take estrogen (900 / 8,000 = 11.25%). Thus, a case-control study with no other biases would be expected to observe an OR of 8.0, whereas the OR for the hypothetical true data is assumed to be 1.0.
- If we examine the selection probabilities, we see that alpha = 1,800 / 2,000 = 0.9, beta = 900 / 8,000 = 0.1125, gamma = 1,600,000 / 2,000,000 = 0.8, and delta = 6,400,000 / 8,000,000 = 0.8. These are shown on the next slide.
- The selection probabilities can be used to assess the presence, direction, and extent of selection bias in the measure of association. For example, in a case-control study, the odds ratio is computed as the odds of exposure among cases divided by the odds of exposure among controls. The odds ratio is computed from the study population, which is assumed to be an unbiased sample of the actual population. So for the odds ratio to be unbiased, the exposure odds in the actual population should be the same as the exposure odds in the target population. (The OR can also be unbiased if both exposure odds are biased in the same direction and by the same [proportionate] amount, in which case the biases offset one another.) The bias in the exposure odds in cases is equal to the ratio of the selection probabilities in cases, alpha/beta. The bias in the exposure odds in controls is equal to the ratio of the selection probabilities in controls, gamma/delta. So in the detection bias scenario we have just presented, alpha/beta = 0.9 / 0.11 = 8, indicating a substantial bias in the observed exposure odds in cases. Since the exposure odds in controls is unbiased (gamma / delta = 0.8/0.8 = 1), the bias in the odds ratio will be the same as the bias in the exposure odds in cases. Since the true OR of 1.0, the observed OR is 8.0
- Interestingly, however, the remedy proposed by Horwitz and Feinstein involved the controls rather than the cases. Arguing that there were undetected cases of endometrial cancer in the population, Horwitz and Feinstein recommended that the control group be recruited from among women who had undergone the same diagnostic procedure for endometrial cancer (i.e., D&C) as had the cases, so that any endometrial cancers among the controls would be detected and could be added to the cases. This proposal has a superficial appeal, since it makes it appear that the cases and controls are being treated the same (recall that the conventional understanding of the case-control study emphasized making a “fair” comparison of cases and controls). However, only a very small proportion of controls would have undetected endometrial cancer, and unless the D&C were being carried out specifically for the study, these women would already have been counted as cases. So the impact on the case group was negligible. In contrast, there was a substantial effect on the control group, since instead of being selected from the general population, to provide an estimate of estrogen use in the population from which cases arose, the control group provided an estimate of estrogen use in a population of women who attended gynecology clinics for evaluation of symptoms. These women were more likely to be receiving postmenopausal estrogens, of course, since evaluation of bleeding and other side effects would be a reason for getting a D&C. In terms of selection probabilities, we might guess that if 6% of postmenopausal women in the general population attend a gynecology clinic, then 36% of postmenopausal women using estrogens attend a gynecology clinic. As a result, gamma/delta now equals 6, and the OR is 1.3.