Your SlideShare is downloading. ×
Critical appraisal of randomized clinical trials
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Critical appraisal of randomized clinical trials

1,870
views

Published on

Published in: Health & Medicine, Business

1 Comment
2 Likes
Statistics
Notes
  • There's a great course to teach this: www.NewcastleFCEM.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,870
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
85
Comments
1
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Streptomycin and bed­rest (S case) or by bed­rest alone (C case).
  • Without Bradford Hill, randomisation would have come about sooner or later, perhaps introduced by Rutstein in the United States. Rutstein collaborated with Bradford Hill in the design of an Anglo­American trial of adrenocorticotrophic hormone, cortisone, and aspirin in the treatment of acute rheumatic fever.
  • Probably approaches 20.000/year
  • A sample of the population of interest is randomly allocated to one or another intervention and the two groups are followed up fora specified period of time. Apart from the interventions being compared, the two groups are treated and observed in anidentical manner. At the end of the study, the groups are analysed in terms of outcomes defined at the outset. The results from, say, the treatment A group are compared with results from the treatment B group. As the groups are treated identically apart from the intervention received, any differences in outcomes are attributed to the trial therapy.In paralel design: each group exposed only to one of study interventions.
  • People behave differently when included in a trial (Hawthorne effect)Employees of Hawthorne Works of Western Electric Company in Chicago participated in a study to evaluate effect of light levels on work performance.Surprisingly, work performance increased, regardless of whether level of light at workplace was increased, kept constant, or decreased.Special attention given to workers participated in the study explains improvement in overall performance
  • RCTs are ‘experiments’ because the investigators can influence the number and the type of interventions, as well as the regimen(amount, route, and frequency) with which the interventions are applied to the participants. This is in contrast to other types of studies, called ‘observational’, in which the events are not influenced by the investigators.
  • Layman: شخص عادي
  • When the sample size of a study is too small, it may be impossible to detect any true differences in outcome between the groups. Such a study might be a waste of resources and potentially unethical. Frequently, however, small sized studies are published that claim no difference in outcome between groups without reporting the power of the studies.Researchers should ensure at the planning stage that there are enough participants to ensure that the study has a high probability of detecting as statistically significant the smallest effect that would be regarded as clinically important.
  • Inference: استدلال استنتاج
  • In clinical research, hypothesis testing risks two fundamental errors. First, researchers can conclude that two treatments differ when, in fact, they do not. This type I error (α) measures the probability of making this false-positive conclusion. Conventionally, α is most frequently set at 0.05, meaning that investigators desire < 5% chance of making a false-positive conclusion.Second, researchers can conclude that two treatments do not differ when, in fact, they do—ie, a false-negative conclusion. This type II error (β) measures the probability of this false-negative conclusion. Conventionally, investigators set at 0.20, meaning that they desire < 20% chance of making a false-negative conclusion.Usually, we recommend estimating the event rate in the population and then determining a treatment effect of interest. For example, investigators estimate an event rate of 10% in the controls. They then would estimate an absolutechange (eg, an absolute reduction of 3%), a relative change (a relative reduction of 30%), or simply estimatea 7% event rate in the treatment group.If only small improvements in the outcome measurements between groups are expected, as may be the case for many chronic diseases, or if the expected outcome occurs infrequently in either group, then a large sample size will be required before these differences achieve statistical significance.
  • The larger the sample size, the more likely randomization will achieve its goal of prognostic balance.Known prognostic factors:Patient’s ageSeverity of illnessComorbidityHost of other factorsUnknown prognostic factors: Although we will never know whether similarity exists for the unknown prognostic factors, we are reassured when the known prognostic factors are well balanced.
  • Concealment: إخفاء - كتمان
  • Concealment: إخفاء – كتمان
  • Conceal: يكتم – يخفي
  • Anyone of the many people who are involved in a trial can distort its results, if they know the identity of the intervention while it is administered or assessed.Moreover, unlike allocation concealment, blinding is not always appropriate or possible. For example, in a RCT where one is comparing enteral nutrition with corticosteroids in the treatment of children with active Crohn’s disease, it may be impossible to blind participants and health care professionals to assigned intervention, although it may still be possible to blind those analyzing the data, such as statisticians.
  • Patients: Patients who are aware that they are receiving what they believe to be an expensive new treatment may report being better than they really are. Doctors: The judgment of a doctor who expects a particular treatment to be more effective than another may be clouded in favor of what he perceives to be the more effective treatment. Analysts: When people analyzing data know which treatment group was which, there can be the tendency to ‘‘overanalyze’’ the data for any minor differences that would support one treatment. It is important for authors of papers describing RCTs to state clearly whether participants, researchers, or data evaluators were or were not aware of assigned treatment.
  • Confusing: يربك - يشوش
  • The terms blinding and masking are used interchangeably.Blinding is not always appropriate or possible. For example, in a RCT where one is comparing enteral nutrition with corticosteroids in the treatment of children with active Crohn’s disease, it may be impossible to blind participants and health care professionals to assignedintervention, although it may still be possible to blind those analyzing the data, such as statisticians.
  • Endpoint:A clearly defined outcome associated with an individual subject in clinical research. Outcomes may be based on safety, efficacy, or other study objectives (eg, pharmacokinetic parameters). An endpoint can be quantitative (eg, systolic blood pressure, cell count), qualitative (eg, death, severity of disease), or time-to event(eg, time to first hospitalization from randomization).ACS: Acute coronary syndrome
  • Surrogate: بديل
  • Patients who are lost often have different prognoses from those who are retained:- they may disappear because they have adverse outcomes - or because they are doing well and so did not return for assessment.
  • However, after randomisation, it is almost inevitable that some participants would not complete the study for whateverreason. Participants may deviate from the intended protocol because of misdiagnosis, non-compliance, or withdrawal.When such patients are excluded from the analysis, we can no longer be sure that important baseline prognostic factorsin the two groups are similar. Thus the main rationale for random allocation is defeated, leading to potential bias.To reduce this bias, results should be analyzed on an ‘intention to treat’ basis.
  • Interim: مؤقت
  • PAD: peripheral arterial diseaseReference: Key topics in EBM, oxford, 2001. What clinicians and patients require is evidence that the treatments improve outcomes that are important to patients (patient-important outcomes), such as avoiding hospitalization for heart failure, or decreasing the risk of myocardial infarction.
  • Odds ratio (OR) and risk ratio (RR):probability of having an event between two groups exposed to a risk factor or treatment.
  • Odds Ratio:we could estimate the odds of having versus not having an event.
  • CI is important because it gives an idea about how precise an estimate is. The width of the interval indicates the precision of the estimate. The wider the interval, the less the precision.A very wide interval may indicate that more data should be collected before anything definite can be said about the estimate.
  • NNT:Number of people who need to receive a treatment in order to achieve the required outcome in one of them.
  • Interim: مؤقت
  • Replete: متخم – مليءSample size calculations only provide sufficient statistical power to test the main hypotheses and need to be multiplied by the number of levels in the subgroups in order to provide this additional power.We encourage you to be skeptical of subgroup analyses. The treatment is likely to benefit the subgroup more or less than the other patients only if the difference in theeffects of treatment in the subgroups is large and very unlikely to occur by chance. Even when these conditions apply, the resultsMay be misleading if investigators did not specify their hypotheses before the study began, if they had a very large number of hypotheses, or if other studies fail to replicate the finding.
  • No plausible biological explanation for this observationGemini: الجوزاءLibra: الميزان
  • Libra:الميزان Gemini: الجوزاء
  • Critical appraisal: Application of rules of scientific evidence to assess the validity of the results of a study. Validity: Extent to which an instrument accurately measures what we want it to measure.
  • Validity: صلاحية
  • Of all the aspects of a trial that have been used to define and assess quality, internal validity is the least context dependent and, as far as we know, the only one that has been the subject of the few empirical studies available.Because of this, we strongly recommend that any assessment of the quality of a trial includes elements related to internal validity.
  • John R Darsee M.D., 34-year-old clinical investigator. One of his paper was published in 1981 in the New England Journal of Medicine. More ink has been shed on Darsee’s case than on any other because: - it was the first major publicised case that was not an isolated blemish on the face of science (not mad – rather, bad)- it concerned prestigious institutions, co-authors, and journals- the charismatic personality of one of the central figures- it started the whole debate about the rights and wrongs of authorship (particularly gift authorship), data retention, the supervision of juniors, and the management of suspected cases of fraud.There was also the realization of the pressure to publish – and not merely important work but anything that showed a department’s activity, though the results should somehow be positive. Finally, the case also shifted the whole climate of feeling of trust to thinking the unthinkable – the possibility that things might not be as they seemed.There was also the new concept: once a crook, often always a crook – Darsee was found to have had a long history of faking his results in different projects and in different settings.
  • 2- Inability of peer review to detect the fraud:Most of Darsee’s work was peer reviewed by good editorial processes and was able to pass without suspicion.4- Focus on the responsibilities & contributions of coauthorsTo be a coauthor requires substantial input into study design, analysis, interpretation, and writing of the research report.
  • Crook: محتال - لص
  • Scales: مقياس
  • The Jadad scale is not the only, or always most appropriate, way to assess trial quality, but it is the most widely used, and appears to produce robust and valid results in an increasing number of empirical studies.
  • 5- Was Follow-up Complete?Patients who are lost often have different prognoses from those who are retained—they may disappear because they have adverse outcomes or because they are doing well and so did not return for assessment.When does loss to follow-up seriously threaten validity?Rules of thumb (you may run across thresholds such as 20%)7- Was the study stopped early?Truncated randomized controlled trials (RCTs) are trials stopped early because of apparent harm,because the investigators have concluded that they will not be able to demonstrate a treatment effect (futility), or because of apparent benefit. Believing the treatment from RCTs stopped early for benefit will be misleading if the decision to stop the trial resulted from catching the apparent benefit of treatment at a random high.
  • Statement: كشف - بيان - إفادةDiagram: مخططChecklist:
  • Transcript

    • 1. Critical appraisal of randomized clinical trials Samir Haffar M.D. Assistant Professor of Gastroenterology
    • 2. Trial design Based on RCTs • Systematic review • Meta-analysis • Randomized controlled trial • Cohort study • Case control study • Cross-sectional study • Case series & case report
    • 3. Hierarchy of evidence in quantitative studies McGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM. BIOS Scientific Publishers, 1st Edition, Oxford, 2001.
    • 4. Perhaps the first large-scale clinical trial using a properly designed randomized schema
    • 5. Sir Austin Bradford Hill (1897-1991) British epidemiologist & statistician The father of modern RCTs
    • 6. Sir Austin Bradford Hill • Studied medicine when World War 1 intervened • Pilot in the World War 1 • Contracted TB: 2 years hospital -2 years convalescence • Took a degree of Economics by correspondence • 1922 Attended statistical lectures by Karl Pearson • 1933 Reader in Epidemiology &Vital Statistics • 1947 Professor of Medical Statistics • 1950-52 President of the Royal Statistical Society
    • 7. First RCT in the United States * Rheumatic Fever Working Party. Circulation 1960 ; 22 : 505 – 15. NIH started a study of adrenocorticotropic hormone (ACTH), cortisone & aspirin in the treatment of rheumatic heart disease* 1951
    • 8. Number of RCT per year Glasziou P, Del Mar C. Evidence based practice workbook. Blackwell Publishing, 2nd edition, 2007. ≈ 20,000 trials published each year > 500,000 trials in total
    • 9. Basic structure of a RCT Parallel trial Akobeng AK. Arch Dis Child 2005 ; 90 : 840 – 844. Parallel trial is the most frequently used design
    • 10. Basics of RCT – 1 • Participants Patients – relatives of pts – healthy volunteers – groups • Investigators People who design & carry out study & analyze results • Interventions Preventive strategies, screening, & treatments Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007.
    • 11.  Placebo Inert pills that appear identical to trial therapy  Gold standard therapy It may be unethical to treat patient with placebo  New treatment Basics of RCT – 2 Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007. Control group should receive one of the following:
    • 12. • Quantitative studies (quantified outcomes) • Most rigorous method of hypothesis testing • Experimental studies versus observational studies • Gold standard to evaluate effectiveness of interventions RCTs are regarded as Basics of RCT – 3 Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007.
    • 13. Some historical examples of treatments with dramatic effects • Insulin for diabetes • Blood transfusion for severe hemorrhagic shock • Defibrillation for ventricular fibrillation • Neostigmine for myasthenia gravis • Tracheotomy for tracheal obstruction • Drainage for pain associated with abscesses • Pressure or suturing for arresting hemorrhage Glasziou P et al. Br Med J 2007 ; 334 : 349 – 351.
    • 14. Parachutes reduce risk of injury after gravitational challenge Their effectiveness has not been proved with RCTs Glasser SP. Essentials of clinical research. Springer, 1st edition, 2008
    • 15. Ethics committee • Include: Layman, religious man, lawyers, researchers & clinicians • Responsibilities: Protect rights & welfare of research subjects Determine if the potential benefits warrant the risks Ensure that informed consent is obtained Prevent unscientific or unethical research
    • 16. The trial team • Principal investigator • Trial coordinator or manager • Trial programmer • Data manager or clerks • Trial statistician Planning phase Interim analyses Final analysis • Trial secretary
    • 17. Randomized controlled trial  Sample size  Randomization  Blinding (Masking)  Outcomes  Intention to treat analysis (ITT)  Measurement of treatment effect  Applicability of results to your patients Critical appraisal
    • 18. Flow diagram for a RCT Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 19. Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.  Sample size in RCTs
    • 20. The “Universe” & the “Sample” Glasser SP. Essentials of clinical research. Springer, 1st edition, 2008
    • 21. Statistical inference Making statistical inferences about a population from a sample by means of significance test & CI Wang D, Bakhai A. Clinical trials: practical guide to design, analysis, & reporting. Remedica, London, UK, 1st edition, 2006.
    • 22. Component of sample size calculation  Type I error (α) False positive = 0.05  Type II error (β) False negative = 0.20 Power (1- β)  Event rate in control group  Event rate in treatment group Schulz KF, Grimes DA. Lancet 2005 ; 365 : 1348 – 53.
    • 23.  Randomization in RCTs If the study wasn‟t randomized we‟d suggest that you stop reading it
    • 24. Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.  Randomization in RCTs If the study wasn‟t randomized we wound suggest that you stop reading it
    • 25. Goal of randomization Comparable groups to known prognostic factors Beta-Blocker Heart Attack Trial - Baseline comparisons Propranolol Placebo (N-1,916) (N-1,921) Average Age (yrs) 55.2 55.5 Male (%) 83.8 85.2 White (%) 89.3 88.4 Systolic BP 112.3 111.7 Diastolic BP 72.6 72.3 Heart rate 76.2 75.7 Cholesterol 212.7 213.6 Current smoker (%) 57.3 56.8 Table comparing baseline characteristics presented in RCT reports
    • 26. Randomization • Simple randomization • Random table • Block randomization • Stratified randomization • Minimization method • Unequal randomization • Allocation concealment Inacceptable Preferred
    • 27. 2 principles of randomization • First They must define the rules that will govern allocation • Second They should follow the same rules strictly throughout the whole study Regardless of the method of randomization used, investigators should follow two principles
    • 28. Simple randomization Inacceptable • Toss of a coin • Date of birth (even numbers to group A) • Hospital admission number • Date seen in clinic Patients seen this week (group A) Those seen next week (group B) Problems arise from openness of allocation system
    • 29. Allocation concealment • Sealed opaque envelope Investigator open several envelopes before allocation Allocation seen if envelope held against bright light • Remote randomization (preferred) Assignment removed from those making assignments: By telephone – Over the internet Randomization should be distant & separate from clinicians conducting the trial
    • 30. RCT of open vs. lap appendectomy • Trial ran smoothly during the day • Surgeon‟s presence required for lap procedure at night • Residents at night held semiopaque envelopes up to light & opened first envelope that dictated open procedure • First eligible patient in the morning allocated to lap group • If patients seen at night sicker than those seen in the day, this behavior bias results against open procedure Hansen J et al. World J Surg. 1996 ; 20 : 17 – 20.
    • 31. Schulz KF et al. JAMA 1995 ; 273 : 408 – 12. Estimates of treatment effect exaggerated by 40% in trials with unconcealed compared with concealed randomization
    • 32.  Blinding in RCTs Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 33.  Blinding /masking in RCTs Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 34. Blinding or masking • Keep one or more of the people involved in the trial unaware of the intervention that is being evaluated • Purpose: decrease risk of observation bias • What matters Not the number of people blinded during a trial But the number & role of those who are not blinded Blinding is not always appropriate or possible
    • 35. • Participants • Investigators who administer interventions • Investigators taking care of the participants • Investigators assessing the outcomes • Data analyst • Investigators who write results of the trial Blinding or Masking Blinding can be implemented in at least 6 levels in RCTs Usually the same
    • 36. Blinding Sometimes called masking • Single blind Only patients or only investigators are ignorant of assigned treatment • Double blind Patients & investigators are ignorant of assigned treatment • Triple blind Patients, investigators & data evaluators are ignorant of assigned treatment
    • 37. Blinding or masking Depending on blinding extent, RCTs classified as • Open label (everyone aware) • Single-blind • Double-blind • Triple-blind • Quadruple-blind & so on
    • 38. The term ‘double-blind RCT’, so often used to represent the ultimate in design to produce valid results, is confusing Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007.
    • 39. Why is blinding so important? • Trials that were not double blinded yielded larger estimates of treatment effects than double blinded trials (OR exaggerated on average by 17%) • Blinding is weaker than allocation concealment in preventing biases Schulz KF. Evid Based Nurs 2000 ; 5 : 36 – 7.
    • 40. A humorous example of blinding/masking Glasser SP. Essentials of clinical research. Springer, 1st edition, 2008
    • 41.  Outcomes in RCTs Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 42. Outcomes in RCTs – 1 • One primary outcome (usually) Most important outcome (stroke in carotid endarterectomy) • Composite outcomes (sometimes – can mislead) - Drug in MI: death, non fatal MI, hospitalization for ACS - Validity depends on similarity in patient importance, treatment effect, & number of events across components - Abandoned if large variations exist between components Montori VM. Br Med J 2005; 330 : 594 – 596. Primary outcome
    • 43. Used in case of rare events of clinical importance Studies in cytoprotection of NSAIDs Endoscopic ulcers surrogates of bleeding or perforated PU Outcomes in RCTs – 2 Surrogate outcomes Secondary outcomes (usually multiple) Other variables important to research question (drugs SE) Too much emphasis if no change in primary outcome
    • 44. Serious GI Events Clinical Ulcers Endoscopic Ulcers Relative Severity GI Symptoms Relative Frequency NSAID-related GI side effects
    • 45.  Intention to treat analysis (ITT) Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 46. Participants who not complete the study • Some participants would not complete the study because of misdiagnosis, non-compliance, or withdrawal • When such patients excluded from analysis, we can no longer be sure that important prognostic factors in the 2 groups are similar which lead to potential bias • To reduce this bias, results should be analyzed on an „intention to treat‟ basis
    • 47. Intention to treat analysis Form of quality control rather than analytic tool • Strategy in conduct & analysis of RCT ensuring that all patients allocated to treatment or control groups analyzed together as representing that treatment arm whether or not they received prescribed therapy or completed study • Randomized participants = Analyzed participants McGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM. BIOS Scientific Publishers, 1st ed, Oxford, 2001.
    • 48.  Measurement of treatment effect Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 49. Measurement of treatment effect in RCTs • p value (p) • Relative Risk (RR) • Odds Ratio (OR) • Confidence Intervals (CIs) • Number Needed to Treat (NNT) Data analyzed as trial proceeds (interim analysis) or at the ends of the trial
    • 50. Probability value (p Value) • p value is probability that observed difference between 2 treatment groups might occur by chance • Many use p value of 0.05 as cut off for significance p < 0.05 Observed difference between groups is so unlikely to have occurred by chance Considered as statistically significant p > 0.05 Observed difference between groups might have occurred by chance Considered as not statistically significant
    • 51. • p > 0.05 Statistically insignificant • p < 0.05 Statistically significant Probability value (p value) Statistically significant Clinically significant Doesn't mean
    • 52. Statistical versus clinical significance • Pentoxifylline vs placebo in PAD* (1992) 40 patients randomized to pentoxifylline or placebo Maximum pain-free walking distance longer in pentoxifylline group than in placebo group (p < 0.001) Conclusion: pentoxiphylline clinically effective • Close examination of data: Difference in maximum walking distance: 3.5 feet Doctors & patients consider it not clinically significant * PAD: Peripheral Arterial Disease McGovern D et al. Key topics in EBM. BIOS Scientific Publishers, Oxford, 2001.
    • 53. Risk & Relative Risk (RR) Number of patients fulfill criteria for a given end point divided by total number of patients i.e.: Diarrhea during tt with antibiotic in 4 of 10 patients Risk of patients: 4 / 10 = 0.4 Diarrhea in control group in 1 of 10 persons Risk of controls: 1 / 10 = 0.1 • Risk Risk of patient / risk of control group RR: 0.4 / 0.1 = 4 • Relative Risk
    • 54. Odds & Odds Ratio (OR) Number of patients fulfill criteria for given endpoint divided by number of patients who do not i.e.: Diarrhea during tt with antibiotic in 4 of 10 patients Odds of patients: 4 / 6 = 0.66 Diarrhea in control group in 1 of 10 persons Odds of controls: 1 / 9 = 0.11 • Odds Odds of patients / odds of control group OR = 0.66 / 0.11 = 6 • Odds Ratio
    • 55. Risk & Odds a a + b Risk a b Odds
    • 56. Interpretation of RR & OR RR or OR should be accompanied by their CIs RR or OR > 1 Increased likelihood of outcome in treatment group RR or OR < 1 Decreased likelihood of outcome in treatment group RR or OR = 1 No difference of outcome between tt & control group
    • 57. Odds ratio or relative risk? OR will be close to RR if endpoint occurs infrequently (<15%) If outcome is more common, OR will differ increasingly from RR Altman DG et all. Systematic reviews in health care: Meta-analysis in context. BMJ Publishing Group, London, 2nd edition, 2001.
    • 58. Significance of CI • When we test a new Crohn‟s disease drug on randomly selected sample of patients, the treatment effect we will get will be an estimate of the „„true‟‟ treatment effect for the whole population of patients with CD in the country • 95% CI of estimate will be range within which we are 95% certain the true population treatment effect will lie
    • 59. Confidence intervals Value 95 % CI are commonly used 90 or 99% CI are sometimes used Width of CI Indicates precision of the estimate Wider the interval, less the precision CI includes 1 No statistically significant difference CI doesn‟t include 1 Statistically significant difference
    • 60. Statistical significance & CI (a) Statistically significant , low precision (b) Statistically significant, high precision (c) Not statistically significant, low precision (d) Not statistically significant, high precision Glasziou P et al. Evidence based practice workbook. Blackwell, 2nd edition, 2007.
    • 61. Influence of sample size on CI precision Width of CI (precision of the estimate) decreases with increasing sample size Peat JK, et al. Health science research. Allen & Unwin, Australia, 1st ed, 2001.
    • 62. Confidence interval or p value? • Authors of articles could report both p values & CIs • CI convey more useful information than p values • If only one is to be reported, then it should be the CI • p value is less important & can be deduced from CI
    • 63. Number Needed to Treat (NNT) • Relative Risk (RR) Risk in treatment group / risk in control group • Relative Risk Reduction (RRR) 1 – RR • Absolute Risk Reduction (ARR) Risk in control group – risk in treatment group • NNT (expressed in clinically relevant way) 1 /ARR
    • 64. • p value (p) • Relative Risk (RR) • Odds Ratio (OR) • Confidence Intervals (CIs) • Number Needed to Treat (NNT) Measurement of treatment effect in RCTs
    • 65. Subgroup analysis Post-hoc analysis • In large trials not demonstrating overall favorable trend, it is common to conduct subgroup analyses to find one or more subgroups in which treatment “really works” • Literature is replete with unconfirmed subgroup findings • Post-hoc results should be regarded as inconclusive • May be of value for hypothesis generation
    • 66. ISIS-2 trial - Subgroup analysis • Effects of streptokinase &/or aspirin on short-term mortality in patients admitted with AMI • Mortality benefits for both active interventions • In subgroup analyses: – Patients born under Zodiac signs of Gemini & Libra 5% higher mortality on aspirin vs placebo – Patients born under other Zodiac signs 30% lower mortality on aspirin vs placebo Sleight P. Curr Control Trials Cardiovasc med. 2000;1(1):25-27.
    • 67. ISIS-2 trial Streptokinase &/or aspirin on AMI mortality Furberg B. Evaluating clinical research. Springer, NY, USA, 2007.
    • 68. It is very difficult to make a judgment if statistics used in a study are appropriate & applied correctly
    • 69. Furberg BD & Furberg CD. Evaluating clinical research. Springer Science & Business Media , 1st ed, 2007.
    • 70. Wang D, Bakhai A. Clinical trials: practical guide to design, analysis, & reporting. Remedica, London, 1st Edition, 2006. Basic understanding of medical statistics will enable us to detect the more obvious errors
    • 71.  Applicability of results to your patients Attia J & Page J. Evid Based Med 2001 ; 6 : 68 – 69.
    • 72. External validity Applicability of results to your patients Issues needed to consider before deciding to incorporate research evidence into clinical practice * Guyatt G, et al. User‟s guide to the medical literature. Essentials of evidence based clinical practice. Mc Graw Hill, 2nd edition, 2008. • Similarity of study population to your population • Benefit versus harm • Patients preferences • Availability • Costs
    • 73. The Trial patients The trial report The actual patients The problem of applying trial results
    • 74. Critical appraisal of a RCT Glasziou P et al. BMJ 2004 ; 328 : 39 - 41.
    • 75. Furberg BD & Furberg CD. Evaluating clinical research. Springer Science & Business Media – First Edition – New York – 2007.
    • 76. Internal & external validity of a RCT Attia J & Page J. Evid Based Med 2001 ; 6 : 68 - 69.
    • 77. • Internal validity of a trial – Randomization – Blinding (Masking) – Follow-up – Outcomes – Analysis – Biases Critical appraisal of a RCT • External validity of a trial (generalizability) – Applicability of results to your patients
    • 78. Benefit versus harm “All that glisters is not gold” W. Shakespeare In “The Merchant of Venice” Furberg BD & Furberg CD. Evaluating clinical research. Springer Science & Business Media – First Edition – New York – 2007.
    • 79. Bias • Difference between the study results & the truth • Of course, we can never know the truth, but we try to come as close as possible by performing & using well-designed & well executed studies • Non-systematic bias (random error or chance) Occurs to similar extent in all subjects for both group Predictable – Less important than systematic bias • Systematic bias (non-random error) Most serious type of bias: under or over-estimation * Guyatt G, et al. User‟s guide to the medical literature. Essentials of evidence based clinical practice. Mc Graw Hill, 2nd edition, 2008.
    • 80. Main types of biases in RCTs Biases Types During planning phase of a RCT Choice-of-question bias Regulation bias Wrong design bias Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007. During course of a RCT Selection bias Observation bias Population choice bias Intervention choice bias Control group bias Outcome choice bias During reporting of a RCT Withdrawal bias Selective reporting bias Fraud bias
    • 81. Fraud bias John Darsee (Harvard researcher in cardiology) • Fabricated data in a study on dogs in 1981 • Fabricated data during his: - Undergraduate days [Notre Dame University, (1966-70)] - Residency & fellowship [ Emory University, (1974-79)] - Fellowship [Brigham & Women‟s, Harvard, (1979-81)] • > 100 papers & abstracts most in prestigious journals • His coauthors had too little contact with the research Listed over their objections (had been helpful in the past)
    • 82. Lessons learned from the Darsee’s affair  Little can be done to stop unscrupulous scientist even when he collaborates with knowledgeable colleagues  Inability of peer review to detect the fraud  Need for explicit guidelines & oversight for collection, maintenance, & analysis of data in clinical trials  Focus on responsibilities & contributions of coauthors  Misconduct investigations may need to examine a researcher‟s entire work over many years Lock S, Wells F, Farthing M. Fraud & misconduct in biomedical research. BMJ Publishing Group, London, 3rd Edition, 2001.
    • 83. Darsee was found to have had a long history of faking his results in different projects & in different settings One of the lessons learned from Darsee’s case „Once a crook, often always a crook‟ Lock S, Wells F, Farthing M. Fraud & misconduct in biomedical research. BMJ Publishing Group, London, 3rd Edition, 2001.
    • 84. Existing tools to assess trial quality • Several components grouped in Scales Each item scored numerically Overall quality score is generated Checklists Components evaluated separately No numerical scores • Systematic search of literature in 1995 identified 25 scales & 9 checklists for assessing trial quality* * Moher D et all. Controlled clinical trials 1995 ; 16 : 62 – 73.
    • 85. The Jadad scale Scores: 0 - 5 points – Poor quality if ≤ 2 points Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd Ed, 2007.
    • 86. Appraising a RCT (checklist) – 1 Are the results valid? During trial  Was trial blinded & to what extent? At end of trial  Was follow-up complete?  Was ITT principle applied?  Was the trial stopped early? Guyatt G, et al. User‟s guide to the medical literature. Essentials of evidence based clinical practice. Mc Graw Hill, 2nd ed, 2008.  Were the patients randomized?  Was the randomization concealed?  Similar prognostic factors in 2 groups? At start of trial
    • 87. What are the results? 8- How large was the treatment effect? 9- How precise was estimate of treatment effect (CI)? How can I apply the results to patient care? 10- Were the study patients similar to my patient? 11- Were all patient-important outcomes considered? 12- Are the likely treatment benefits worth harm & cost? Guyatt G, et al. User‟s guide to the medical literature. Essentials of evidence based clinical practice. Mc Graw Hill, 2nd ed, 2008. Appraising a RCT (checklist) – 2
    • 88. Scales or checklists? No consensus on which is preferable •CDSR: Cochrane Database of Systematic Reviews Moher D et all. Health Technol Assess 1999 ; 3 (12). Quality assessment in systematic reviews Medical journals CDSR* Number of SR 78 SR in 204 journals 36 SR Checklists 20/78 (26%) 92 % Scales 52/78 (67%) None
    • 89. * Altman DG et al. Ann Intern Med 2001 ; 134 : 663 - 94. Improving quality of reports Quality of Reporting of Meta-analyses Meta-analysis QUOROM** Consolidated Standards of Reporting Trials RCTs CONSORT* Diagnostic accuracy study STARD*** Standards for Reporting of Diagnostic Accuracy ** Moher D et al. Lancet 1999 ; 354 : 1896 - 900. *** Bossuyt PM et all. BMJ 2003; 326 : 41 – 44.
    • 90. CONSORT statement Targeted authors of trial reports rather than readers • Experts Clinical epidemiologists, journal editors, & biostatisticians published CONSORT statement • Aim Improve standard of written reports of RCTs • Results Latest version of CONSORT statement includes2 Flow diagram: Patients progress through a trial Checklist: 22 items 1 Begg C, et all. JAMA 1996 ;276 (63): 7 – 9. 2 Moher D, et al. CMAJ 2004 ; 171 : 349 – 350.
    • 91. Flow diagram of a RCT Ann Intern Med 2001 ; 134 : 657 – 662.
    • 92. CONSORT statement Ann Intern Med 2001 ; 134 : 657 - 662. Paper Section & Topic Item Descriptor Reported on Page No Title & abstract 1 How participants allocated to interventions Introduction background 2 Scientific background Methods Participants Interventions Objectives Outcomes Sample size Randomization Blinding (masking) Statistical methods 3 4 5 6 7 8-9-10 11 12 Criteria for participants, settings, locations Details of interventions for each group Specific objectives & hypotheses Defined primary & secondary outcomes How sample size was determined? Allocation concealment, implementation Whether or not blinding applied Statistical methods used Results Participant flow Recruitment Baseline data Numbers analyzed Outcomes, estimation Ancillary analyses Adverse events 13 14 15 16 17 18 19 Flow diagram strongly recommended Periods of recruitment & follow-up Baseline characteristics of each group No of participants in each group Summary of results with 95% CI Subgroup & adjusted analyses All important adverse events Comment Interpretation Generalizability Overall evidence 20 21 22 Interpretation of the results External validity of trial findings General interpretation of results
    • 93. Reasons for doing RCTs • Only study design that can prove causation • Required by FDA (and others) for new drugs and some devices • Most influential to clinical practice
    • 94. Disadvantages of RCTs • Expensive: typically in $ millions • Time consuming: typically years • Can only answer a single question • May not apply to some patients in practice • May not be practical • Generally difficult to get funded • Organizationally complex
    • 95. Carefully conducted observational studies may provide more evidence than poor RCTs* * Guyatt G, et al. User‟s guide to the medical literature. Essentials of evidence based clinical practice. Mc Graw Hill, 2nd edition, 2008. ** Jadad AR, Enkin MW. Randomized control trials. Blackwell Publishing, 2nd ed, 2007. Unfortunately, a perfect trial can only exist in our imagination**
    • 96. High quality/relevant data – Pearls Glasziou P, Del Mar C & Salisbury J. Evidence based medicine Workbook. BMJ Publishing Group, 1st ed, London, 2003. Pearls selected from the rest of lower quality literature
    • 97. Mc Graw Hill 2008 Blackwell Publishing 2007 References John Wiley & Sons 2006
    • 98. Thank You