Upcoming SlideShare
×

# Critical Appriaisal Skills Basic 1 | May 4th 2011

2,590 views

Published on

NHS Education Scotland
Produced in collaboration with the
Association of Scottish
Medicines Information
Pharmacists Group

Published in: Health & Medicine, Business
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
2,590
On SlideShare
0
From Embeds
0
Number of Embeds
1,326
Actions
Shares
0
35
0
Likes
0
Embeds 0
No embeds

No notes for slide
• Today we are going to concentrate on appraising the RCT.
• Research starts with 1 of 2 hypothesis to be tested
• Say before starting on slide contents: Having explained what ‘p’ is in isolation, we should consider its application to research papers. When making a statement that a difference has been found between the experimental group and the control group in a study, the ‘p’ value that follows is …… Say after finished with slide contents: So if the statement appears Drug A lowers BP more than Drug B, it means “ we think Drug A lowers BP more than Drug B but there is still a 1 in 20 chance we are wrong.
• The smaller the effect we are trying to measure the more subjects/observations we need in the sample The we set the level of significance, the larger the sample we need to detect an event of a given size
• Type 1 &amp; Type 2 errors – probability of occurrence decreases as sample size increases Type 2 Error sometimes used by unscrupulous drug companies when marketing ‘me too’ drugs and making comparison to the market leader-  using a smaller number of patients in each group is unlikely to show a ‘statistically’ difference between the two medications.
• When a study is carried out in a particular population, for practicality purposes only a sample from this population can be chosen. Different types of sampling techniques used (e.g. random, quota, stratified, etc) to try to ensure that the sample selected is representative of the population being studied. However, do not know if the sample represents the population. If poor sampling then variation may arise. Hence, means of estimating the representativeness of the population is required
• It is neither standard nor does it represent error! To understand its correct usage lets imagine: Take a sample of people from a large population -&gt; measure particular parameter and take the mean. Then replace the sample and repeat the process. After measuring many you would have a collection of the means, which when plotted would produce a normal distribution. The mean of these means would then be the ‘true population mean’ and the standard deviation of these means is the famous ‘Standard Error of the Means’
• Relating percentages to clinical practice is difficult NNT is easier to understand and apply. Example = Therefore, 14 patients must be treated with drug X to prevent one death at five years.
• ### Critical Appriaisal Skills Basic 1 | May 4th 2011

1. 1. Critical Appraisal Skills Basic I <ul><li>NHS Education Scotland </li></ul><ul><li>Produced in collaboration with the </li></ul><ul><li>Association of Scottish </li></ul><ul><li>Medicines Information </li></ul><ul><li>Pharmacists Group </li></ul>
2. 2. What is critical appraisal? <ul><li>This is the term given to describe the skills used when reading a paper to enable one to assess the validity (i.e. how close to truth) and usefulness (i.e. can the results be applied to your practice) of the results. </li></ul><ul><li>Forms an integral part of evidence based medicine (EBM). </li></ul>
3. 3. What is EBM? <ul><li>EBM is the judicious use of current best evidence, combined with clinical experience, to make decisions about patient care. </li></ul>
4. 4. 5 key steps that underpin EBM <ul><li>Define the specific question to be answered. </li></ul><ul><li>Find the best evidence to answer the question. </li></ul><ul><li>Critically evaluate the evidence to assess it validity and usefulness. </li></ul><ul><li>Apply the results of the critical evaluation to practice. </li></ul><ul><li>Evaluate the performance of the intervention. </li></ul>
5. 5. Why do we need evidence? <ul><li>Resources should only be allocated to interventions that are effective. </li></ul><ul><li>The only way of judging effectiveness is EVIDENCE! </li></ul>
6. 6. What are “good” sources of evidence? <ul><li>Less reliable sources </li></ul><ul><li>Glossy literature from pharmaceutical companies </li></ul><ul><li>Press releases from pharmaceutical companies </li></ul><ul><li>Magazines such as Pulse </li></ul><ul><li>Advertisements in medical journals </li></ul><ul><li>Conference abstracts about clinical trials </li></ul><ul><li>Trusted sources </li></ul><ul><li>Scottish Medicines Consortium </li></ul><ul><li>SIGN </li></ul><ul><li>NICE </li></ul><ul><li>National Library for Health </li></ul><ul><li>Peer reviewed clinical journals </li></ul><ul><li>Summaries of information published by NHS bodies (e.g. National Prescribing Centre) </li></ul>
7. 7. Hierarchy of evidence <ul><li>Evidence comes in different forms and can be ranked in terms of importance. </li></ul><ul><li>Quite often there may not be any high-levels of evidence to support a clinical intervention. In these cases it may be necessary to use evidence from the lower end of the hierarchy scale. </li></ul>
8. 8. Expert opinion/ clinical experience from respected sources Case series then Case reports Cross sectional surveys Case control studies Cohort studies RCTs Systematic reviews Meta-analyses Hierarchy of evidence
9. 9. Format of clinical trials <ul><li>Clinical trials are usually written in a standard format. This normally consists of: </li></ul><ul><ul><ul><ul><ul><li>Title </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Authors </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Abstract – contains a brief summary of the trial. </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Introduction </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Methods </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Results </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Discussion </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Conclusion </li></ul></ul></ul></ul></ul>
10. 10. Key questions to ask when assessing how valid and useful a clinical trial is <ul><li>Patient or population – does the trial assess a commonly seen clinical condition and were the patients studied similar to your patient? </li></ul><ul><li>Intervention – what was the medicine being tested? Was it used at the normal dose? </li></ul><ul><li>Comparison – What was the experimental medicine compared to? Trials that compare a new medicine to a placebo are not useful when trying to decide where the medicine fits into current practice. </li></ul><ul><li>Outcome – Was the end-point relevant to the patient (i.e. was it patient-oriented like a reduction in risk for a having a heart attack)? </li></ul>
11. 11. Outcomes <ul><li>Patient orientated </li></ul><ul><ul><li>Outcomes that directly improve the outcome for patients. </li></ul></ul><ul><ul><li>e.g. Reduction in hip fracture; improved cardiovascular mortality; prevention of a stroke </li></ul></ul><ul><li>Disease orientated </li></ul><ul><ul><li>Outcomes that are the result of a change to a disease characteristic </li></ul></ul><ul><ul><li>e.g. Change in bone mineral density; lowering of blood pressure; a reduction in cholesterol </li></ul></ul>
12. 12. Surrogate outcomes <ul><li>Outcomes that are not patient-orientated are surrogate outcomes. </li></ul><ul><li>Often physiological or biochemical markers </li></ul><ul><li>Cannot always assume that they are always a good indication of disease progression or improved survival. </li></ul><ul><ul><li>For example a modest decrease in blood pressure may not be clinically relevant even though the result may be statistically significant. </li></ul></ul>
13. 13. Three key factors affecting the results of a trial <ul><li>The intervention </li></ul><ul><li>Bias – researchers take steps to minimise by use of a control group, randomisation and blinding but some biases can still exist. </li></ul><ul><li>Chance – statistical tests are used to assess this. </li></ul>
14. 14. Types of Statistics <ul><li>Descriptive </li></ul><ul><ul><li>Summarises or describes the sample </li></ul></ul><ul><ul><li>[Please note that for purposes of critical appraisal this type will not be discussed in this training package]. </li></ul></ul><ul><li>Inferential </li></ul><ul><ul><li>Concerned with generalising from the sample to make inferences and estimates about a wider population. </li></ul></ul>
15. 15. Inferences and Estimates <ul><li>Inferences </li></ul><ul><ul><li>Can conclusions be drawn from the sample be generalised to the population? </li></ul></ul><ul><ul><ul><ul><ul><li>e.g. If a better response is seen with a medicine in a sample will it hold true in the population </li></ul></ul></ul></ul></ul><ul><ul><li>Help answer whether results may have occurred by chance in the trial. </li></ul></ul><ul><li>Estimates </li></ul><ul><ul><li>Given an observed size of effect in the sample, what is the likely value (or range of values) you will see in the population? </li></ul></ul><ul><ul><li>Help assess usefulness of a trial. </li></ul></ul>
16. 16. Inferential Statistics
17. 17. <ul><li>There are 1 of 2 assumptions made for interventions in clinical trials: </li></ul><ul><ul><li>Null hypothesis (i.e. no difference between the control group and the experimental group). </li></ul></ul><ul><ul><li>Alternative hypothesis (i.e. there is a difference between the control group and the experimental group). </li></ul></ul><ul><li>Generally it is the null hypothesis that is assumed however. </li></ul>Hypothesis Testing
18. 18. <ul><li>The probability that a difference will be seen between 2 interventions in a clinical trial. </li></ul><ul><li>Measured on a scale of 0 (impossible for event to happen) to 1 (the event will certainly happen) </li></ul><ul><ul><ul><li>i.e. P = 1 would always happen </li></ul></ul></ul><ul><ul><ul><li>P = 0.05 would happen 1 time in 20 </li></ul></ul></ul><ul><ul><ul><li>P = 0.02 would happen 1 time in 50 </li></ul></ul></ul><ul><ul><ul><li>P = 0.01 would happen 1 time in 100 </li></ul></ul></ul><ul><ul><ul><li>P = 0.001 would happen 1 time in 1000 </li></ul></ul></ul>Probability (P)
19. 19. <ul><li>If p-value is less than 1 in 20 (p<0.05) then the result is regarded as being statistically significant; and the possibility of the difference observed arising by chance is low </li></ul><ul><li>=> If this is the case then one can reject </li></ul><ul><li> the null Hypothesis </li></ul>P values
20. 20. <ul><li>The probability that a test will detect a real difference in treatment outcomes in a sample if it is present in the population </li></ul><ul><li>Usually expressed as a percentage and often set at 80-90% </li></ul>Power
21. 21. Power and Sample Size <ul><li>Sample size determinants: </li></ul><ul><ul><li>Size of the difference being investigated </li></ul></ul><ul><ul><li>Level of significance </li></ul></ul>
22. 22. Errors that can arise when drawing conclusions from data <ul><li>Type 1 error (alpha) </li></ul><ul><ul><ul><li>The data suggests a difference between the groups when there is really no difference = False positive </li></ul></ul></ul><ul><ul><ul><li>Often called significance level </li></ul></ul></ul><ul><ul><ul><li>A level of significance of p<0.05 represents a 5% probability of making a type 1 error </li></ul></ul></ul>
23. 23. Errors that can arise when drawing conclusions from data <ul><li>Type 2 error (beta) </li></ul><ul><ul><li>The results fail to pick up a real difference that exists between the groups, and a conclusion is made that no difference exists = False negative </li></ul></ul><ul><ul><li>100-(power)% is the probability of making a type 2 error </li></ul></ul>
24. 24. Population Estimates
25. 25. <ul><li>A tool for inferring the characteristics / parameters of a whole population from the measurements in one sample </li></ul><ul><li>One of the most widely misused terms in statistics </li></ul><ul><li>In 95% of cases the ‘True Population Mean’ will lie within +/-2 SEM of the sample mean </li></ul><ul><li>Should NEVER be used instead of SD to indicate dispersion of measurements </li></ul><ul><ul><ul><ul><ul><li>SEM = SD </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>------ </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li> n </li></ul></ul></ul></ul></ul>Standard Error of the Mean (SEM)
26. 26. <ul><li>Represents the range of values within which the true population mean lies. </li></ul><ul><li>Indicate the precision (or imprecision) with which a study sample estimates the true population value for the whole population. </li></ul><ul><li>Important role whenever we wish to apply results of a clinical study to the general population </li></ul><ul><li>Narrower the range the more reliable the results </li></ul>Confidence Intervals (CI)
27. 27. Confidence Intervals <ul><li>Calculated by adding and subtracting multiples of the standard error of the mean to and from the sample mean </li></ul><ul><li>95% confidence interval normally used (i.e. can be 95% confident that the population value lies within this interval); or alternatively stated that there is a 1 in 20 chance (5%) that the true value lies outside the range quoted. </li></ul><ul><li>The narrower the CI, the more confident you can be the sample represents the population </li></ul>
28. 28. <ul><li>Comparing means: </li></ul><ul><ul><li>No difference if CI overlap (i.e. even though 2 mean values may differ, extensive overlap of their respective CIs may suggest that the difference is not statistically significant) </li></ul></ul><ul><li>When comparing differences between means: </li></ul><ul><ul><li>No difference if CI includes 0 </li></ul></ul><ul><li>For proportions (e.g. RR): </li></ul><ul><ul><li>No differences if CI includes 1 </li></ul></ul><ul><ul><li>For further information on this topic: </li></ul></ul><ul><ul><li>-Statistics in divided doses: Number 8 (July 2005).Produced by the North West Medicines Information Service. Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf </li></ul></ul>How to Interpret?
29. 29. Estimation Statistics Help assess “usefulness of the trial” by determining clinical importance and magnitude of the benefit by using data to estimate a range of probable values for the population.
30. 30. Example study 96 1128 Control Group (Received Placebo) 845 2073 Intervention Group (Received Drug X) Pain free within 2 hours Total Number of patients in each group Group
31. 31. <ul><li>Definition: </li></ul><ul><ul><li>The proportion of patients in whom an event is observed </li></ul></ul><ul><li>Control Event Rate (CER) </li></ul><ul><li>Vs </li></ul><ul><li>Experimental Event Rate (EER) </li></ul>Event Rates
32. 32. <ul><li>Control Event Rate (CER) = </li></ul><ul><ul><li>Event Rate in control group </li></ul></ul><ul><ul><li>Total patients in control group </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>CER = 96/1128 = 0.085 (9%) </li></ul></ul>Control Event Rate
33. 33. <ul><li>Experimental Event Rate (EER) = </li></ul><ul><ul><li>Event Rate in experimental group </li></ul></ul><ul><ul><li>Total patients in experimental group </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>EER = 845/2073 = 0.41 (41%) </li></ul></ul>Experimental Event Rate
34. 34. <ul><li>Absolute Risk Reduction (ARR) is way of expressing differences between groups. </li></ul><ul><li>It is the difference in the event rate between the control event rate (CER) and the experimental event rate (EER). </li></ul><ul><ul><ul><ul><ul><li>ARR = CER-EER </li></ul></ul></ul></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>ARR = 9-41 = -32 </li></ul></ul>Absolute Risk Reduction
35. 35. <ul><ul><li>Is an alternative means of expressing the difference between groups as a percentage </li></ul></ul><ul><ul><li>The Relative Risk Reduction (RRR) is the percent reduction in events in the experimental event rate (EER) and the control event rate (CER). </li></ul></ul><ul><ul><li>RRR = (CER-EER) X 100 </li></ul></ul><ul><ul><li>CER </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>RRR = (9-41/9) X 100 = 356% </li></ul></ul>Relative Risk Reduction
36. 36. Why calculate? <ul><li>Sometimes the trial may just state “the treatment reduced the risk” but does not state whether this is relative risk reduction or absolute risk reduction. Obviously the relative risk reduction looks more impressive since a larger number. Be aware of this and use the figures given to calculate both. </li></ul><ul><li>Neither RRR or ARR are intuitive ways to look at data. Numbers needed to Treat (NNT) is the more relevant way to look at the figures. </li></ul>
37. 37. <ul><li>Definition: </li></ul><ul><ul><li>The number of people who needed to be treated to produce one particular clinical outcome </li></ul></ul><ul><ul><li>(e.g. How many patients need to receive Drug X instead of placebo to allow one patient to be pain free at 2 hours?) </li></ul></ul><ul><ul><li>NNT = ____1____ or ____1____ </li></ul></ul><ul><ul><li> CER-EER ARR </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>NNT = 1/32 = 3 </li></ul></ul>Numbers Needed to Treat (NNT)
38. 38. Numbers Needed to Harm (NNH) <ul><li>This value can be similarly calculated when looking at adverse effects in a clinical trial. </li></ul><ul><li>It is the number of patients you would need to treat with the experimental medicine (rather than the placebo or control) for one additional patient to suffer an adverse effect. </li></ul><ul><ul><ul><ul><ul><li>NNH = 1/(EER-CER) </li></ul></ul></ul></ul></ul>
39. 39. Example NNH <ul><li>Medicine Y is given to patients for treatment of hypertension. 12 of the 4000 patients given medicine Y experience a rash compared with 2 out of 3000 given placebo. </li></ul><ul><ul><ul><ul><ul><li>CER = 2/3000 = 0.00066 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>EER = 12/4000 = 0.003 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>NNH = 1/ (0.003-0.00066) = 428 </li></ul></ul></ul></ul></ul><ul><ul><li>Therefore 428 patients must be treated with medicine Y </li></ul></ul><ul><ul><li>rather than placebo for an additional 1 patient to have an </li></ul></ul><ul><ul><li>adverse effect. </li></ul></ul>
40. 40. Relative Risk <ul><li>The relative risk (RR) is the size of the effect in the experimental group relative to the size of the effect in the control group. The relative risk is often quoted in a clinical trial paper. </li></ul><ul><ul><li>RR = CER/EER </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><ul><li>RR= 9/41 = 0.21 </li></ul></ul><ul><ul><li>A relative risk of 1.0 means that there is no difference between the experimental and control groups. This result shows a RR < 1.0 indicating that the patients on the medication are more likely to be pain free at 2 hours than those receiving placebo. </li></ul></ul>
41. 41. <ul><li>The ratio of patients in the treatment group succumbing to a particular end point compared to the control group </li></ul><ul><li>Compares the probability of the event occurring with the probability that it will not occur. </li></ul><ul><li>If >1 = event more likely to happen </li></ul><ul><li>If <1 = event less likely to happen </li></ul>Odds Ratio (OR)
42. 42. <ul><li>The odds ratio must be calculated first for control and treatment group: </li></ul><ul><ul><ul><li>Control = 1032/1128 = 0.915 </li></ul></ul></ul><ul><ul><ul><li>Treatment = 1228/2073 = 0.592 </li></ul></ul></ul><ul><ul><ul><li>Odds Ratio = treatment / control = 0.592/0.915 = 0.647 </li></ul></ul></ul><ul><li>In isolation it is difficult to imagine what this figure means but the smaller the odds ratio the more effective Drug X is in allowing a patient to be pain free within 2 hours . </li></ul>Odds Ratio (II)
43. 43. Summary <ul><li>Don’t always believe everything you read! </li></ul><ul><li>Choose your source of evidence wisely and systematically to answer your question. </li></ul><ul><li>Use estimation statistics to help evaluate usefulness and clinical importance of a trial. </li></ul><ul><li>Utilise population estimates to determine how reflective of the true population the trial results are likely to be. </li></ul><ul><li>Statistical significance does not always equate to clinical significance. </li></ul><ul><li>There is a lot of information out available but you have to choose the best evidence available. Remember that all evidence is not equal! </li></ul>
44. 44. References <ul><li>Brignell J. How do Relative Risk and Odds Ratio compare? ( April 2006). Available at http://www.numberwatch.co.uk/rr&or.htm </li></ul><ul><li>Burls A. What is Critical Appraisal in Evidenced Based Medicine 2nd ed. Oxford: University of Oxford. Available at www.whatisseries.co.uk </li></ul><ul><li>DeCaro, S. A. A student's guide to the conceptual side of inferential statistics (2003) . Available from http://psychology.sdecnet.com/stathelp.htm . </li></ul><ul><li>Easton V, McColl JH. Confidence Intervals in Statistics Glossary V1.1. Available from http://www.stats.gla.ac.uk/steps/glossary/confidence_intervals.html#confinterval </li></ul><ul><li>Greenhalgh T. How to read a paper: The basics of evidence based medicine 2nd edition. London: BNJ Books 2001. </li></ul><ul><li>Swinscow TDV, Campbell MJ. Statistics at Square One 10th edition. London: British Medical Association 2002. </li></ul><ul><li>Statistics in Divided Doses, Assessing the reliability of a sample (Number 3). North West Medicines Information Service (September 2001). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose3.pdf </li></ul><ul><li>Statistics in Divided Doses, Variability, probability and power (Number 4). North West Medicines Information Service (May 2002). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose4.pdf </li></ul><ul><li>Statistics in Divided Doses, First steps in analysis - comparing the means of large samples (Number 5). North West Medicines Information Service (November 2002). Available at </li></ul><ul><li>http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose5.pdf </li></ul><ul><li>Statistics in Divided Doses, Confidence intervals (Number 8). North West Medicines Information Service (July 2005). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf </li></ul><ul><li>Wills S et al. Critical Appraisal of Clinical Trials E-learning Module via NHS Education South Central. Available for free registration at http://www.learning.nesc.nhs.uk/ </li></ul>