Introduction to Pharmacovigilance Signal Detection


Published on

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Introduction to Pharmacovigilance Signal Detection

  1. 1. Introduction to Signal Detection by Rodney L. Lemery MPH, PhDCopyright  BioPharm Systems, Inc. 2011. All rights reserved.
  2. 2. Topics… • Definitions – Pharmacoepidemiology – Pharmacovigilance – Signal – Signal Detection • Signal Detection – Qualitative • Striking cases • Periodic reviews – Quantitative • Analysis of disproportionalityIntroduction to Signal Detection 2
  3. 3. Topics… • Signal Detection (cont.) – Disproportionality analysis tools • Signal Prioritization – WHO Triage – MHRA Impact Analysis – Categorization of priority • Confirmed Signal • Unconfirmed Signal • Refuted Signal • Signal Evaluation – Causality – Frequency – Clinical implications – PreventabilityIntroduction to Signal Detection 3
  4. 4. Acronyms and General Definitions • Prevalence – The total number of cases of a disease in a given population at a specific time • Incidence – The number of newly diagnosed cases during a specific time period • EMA – European Medicines Agency • ADR – Adverse Drug Reaction • APR – Adverse Product Reaction • ICH – International Conference on Harmonization • CIOMS – Council for International Organizations of Medical SciencesIntroduction to Signal Detection 4
  5. 5. Definitions… • Two pervasive definitions (Abenhaim, Moore, & Begaud, 1999) 1. The collection and scientific evaluation of adverse drug reactions (ADR), under normal conditions of use for regulatory purpose. −Restricts the concept to regulatory Pharmacovigilance compliance only and only medicinal products. 2. Watchfulness in guarding against danger from products or providing for safety of the product −Expansive beyond just regulations and frames the construct for use in academia and the sciencesIntroduction to Signal Detection 5
  6. 6. Definitions… •The application of epidemiologic techniques used to study the effects of Pharmacoepidemiology drugs in populations −First mentioned in the early 1980’s (Abenhaim, Moore, & Begaud, 1999)Introduction to Signal Detection 6
  7. 7. Exercise 1 1. Please segregate into your Define the small group as assigned by the index card in your chair following terms: 2. Using collaborative discussion, please reach consensus for the definition of the following terms: Signal  “Signal”  “Signal Detection” 3. Nominate a single spokesperson from the group to Signal Detection share the definition with the larger audience.Introduction to Signal Detection 7
  8. 8. Definitions… •Much debate on the definition (we will use the following): •Information that arises from one or multiple sources, which suggests a new potentially causal association, or Signal a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, that is judged to be of sufficient likelihood to justify verificatory actions. (CIOMS, 2010 p.14)Introduction to Signal Detection 8
  9. 9. Definitions •Much debate on the definition (we will use the Signal following): •The act of looking for and/or identifying signals Detection using event data from any source . (CIOMS, 2010 p.116)Introduction to Signal Detection 9
  10. 10. Simplified Safety Signal Lifecycle Signal Signal Detection Prioritization Signal Evaluation CIOMS (2010, p. 9)Introduction to Signal Detection 10
  11. 11. Detailed Signal Management Process Signal Detection Signal PrioritizationSignalEvaluation 11Introduction to Signal Detection
  12. 12. Signal Detection…Introduction to Signal Detection 12
  13. 13. Individual Case Safety Reports (ICSR) • The accumulation of ICSR can occur from multiple places according to the ICH E2D Guideline (ICH, 2003) Sources of ICSR Description of Sources Unsolicited sources Spontaneous reporting Solicited sources Any organized collection of data (outcomes research, clinical trials, registries, surveys, billing databases etc.) Contractual agreements Inter-company exchange of safety data Regulatory authority Any ICSR originating from the regulatory authority submitted to a companyIntroduction to Signal Detection 13
  14. 14. Signal Detection… Traditional Pharmacovigilance Methods · Individual case review · Aggregate analysis · Periodic reports • Traditionally, signals are detected through the assessment of individual case safety reports (ICSR) in an individual or cumulative mannerIntroduction to Signal Detection 14
  15. 15. Signal Detection…• Qualitative review of ICSR – Critical method of detection for events where the background incidence is uncommon or rare and should not be replaced by quantitative methods (Egberts, Meyboom, & van Puijenbroek, 2002; Hopstadius, Norény, Bate, & Edwards, 2008) – May be implemented as a list of designated medical events (DME) within a company • Often manifest in systems as custom Standardized MedDRA Queries (SMQ) – FDA has a list of “Interesting PTs” – EMA follows serious events identified by CIOMS V• Results in the identification of “index” or “striking” cases that can be monitoredIntroduction to Signal Detection 15
  16. 16. Signal Detection… • Qualitative review of ICSR (cont.) – Periodic review of case series • PSUR or equivalent – Review of the aggregate tables displayed in these periodic reports can provide indications of potential “striking” events • Specific sections such as the “Overall Safety Evaluation” provide clinical context around the identification of such signals and become an important part of signal detection (Waller, 2009 p. 67)Introduction to Signal Detection 16
  17. 17. Exercise 2 1. Please segregate into your small group as assigned by the index card in your chair 2. Using collaborative discussion and the DSUR provided, please DSUR AE determine the presence or Summary Table absence of striking cases in the table. Review  Why did you or didn’t you identify cases of interest? 3. Nominate a single spokesperson from the group to share the results with the larger audience.Introduction to Signal Detection 17
  18. 18. Signal Detection… • Quantitative assessment of ICSR – Absolute counts • Number of reports of an adverse event (AE) or adverse product reaction (APR) 985 No APR All completed the 1 year 1000 duration of study resulting in Participants 985 person-years in an IND study Steven’s Johnson Syndrome (SJS) 15 Completed the 6 month mark of study resulting in 7.5 person-yearsIntroduction to Signal Detection 18
  19. 19. Signal Detection… • With a total of 992.5 person-years for the denominator of the study incidence rate, we can calculate the incidence rate of SJS in the IND population as 15/992.5 which is 0.0151 cases/992.5 person years – A typical denominator is million person years so we must convert or IND incidence rate to million person years • In terms of million person years this becomes 15.1 cases/million person years • Using epidemiologic sources, we note that the estimated background incidence rate of SJS is 1-6 cases/million person years • The incidence rate in the study is suspiciously higher than the expected background rate therefore SJS is a signalIntroduction to Signal Detection 19
  20. 20. Signal Detection… • Quantitative assessment of ICSR (cont.) – Proportions • Number of specific AE reports divided by total number of reports for a given product – Assume an IND study of 100 patients for a new Anticonvulsant – At the end of the study there are 450 ICSR for the product – Within this 450 are 25 reports of “GI bleeding” – The proportion would be 25/450 or ~6% • NOTE: True “expected” proportions are not really possible to determine from spontaneous reporting data – However, highly prevalent product/event combinations for given case series may be enough to elevate the combination to a signalIntroduction to Signal Detection 20
  21. 21. Exercise 3 Determine the following with the 1. Please segregate into your small group as assigned by the data distributed: index card in your chair 2. Using collaborative discussion and the information provided, Is there an please calculate the study prevalence for “Peripheral indication that neuropathy” and compare it to “Peripheral the provided epidemiologic data neuropathy” may  Has a signal been identified in the data? have a relationship 3. Nominate a single to the spokesperson from the group to consumption of share the results with the larger audience. “Alfenta”?Introduction to Signal Detection 21
  22. 22. Disproportionality… Data Mining Algorithms · Disproportional reporting ratios • Once the ICSR database becomes large enough*, statistical techniques (generally referred to as data mining) can be applied – Usually on large datasets from regulatory agencies or public health entities – WHO: Vigibase – FDA: AERS – MCA: Sentinel (formally known as ADROIT) – EMA: EudraVigilance – Generally these techniques identify disproportionate reporting ratios *”Large Enough” is a function of product/event incidence in the populationIntroduction to Signal Detection 22
  23. 23. “Large Enough” • Implications on population size for sampling in signal detection can be reduced to the following (CIOMS, 2010 p.31) : Event Background Example Ease of Signal Detection Incidence Incidence proving an Method in Product of Event association Takers (method) Common Rare Phocomelia Easy ICSR or Periodic due to Thalidomide (clinical observation) Review Rare Rare Reye’s syndrome Less easy ICSR or Periodic and Aspirin (clinical observation) Review Common Common Cough Difficult Data Mining and ACE inhibitors (large observational trials/data) Uncommon Common Breast carcinoma Very difficult Data Mining to Rare and Hormone (large clinical trials) Replacement Therapies Rare Common None known Virtually impossible Virtually impossibleIntroduction to Signal Detection 23
  24. 24. Classical Versus Bayesian… • The basis of disproportionality analysis is either frequentist statistics or Bayesian (Gajewski & Simon 2008) – Classical (frequentist) statistics look at probabilities as long term frequency with an assumption of repeatable experiment or sampling methods and a “true” value for a parameter • TOTAL INFORMATION=Data from experimentation – Bayesian statistics look at “true” probabilities as a function incorporating prior beliefs or knowledge and is updated • TOTAL INFORMATION=Historical Information + Data from experimentationIntroduction to Signal Detection 24
  25. 25. Classical Versus Bayesian… (Maggid, 2011)Introduction to Signal Detection 25
  26. 26. Data Mining Algorithms… • All measures calculated from a 2X2 Contingency Table – Classical • Proportional Rate Ratio (PRR) • Reporting Odds Ratio (ROR) • Relative Reporting Ratio (RRR) – Bayesian • Information Component • MGPS/EBGM Event of All other Events TOTAL Interest Product of Interest A B A+B All other Products C D C+D TOTAL A+C B+D A+B+C+DIntroduction to Signal Detection 26
  27. 27. Data Mining Algorithms… • All measures of disproportionate reporting are basically calculations of OBSERVED/EXPECTED • In PV, the EXPECTED data is also referred to as the “background” • What you include in the “background” is a point of contention in the industry and no real rules are present (Gogolak, 2003)Introduction to Signal Detection 27
  28. 28. Data Mining Algorithms… • Since the simple calculation is O/E, the relationship between background and the statistic of interest is inversely related: – As the background increases the resulting statistic decreases • Large E results in small PRR – As the background decreases the resulting statistic increases • Small E results in large PRRIntroduction to Signal Detection 28
  29. 29. Data Mining Algorithms • In general, the frequentist or Bayesian methods will perform similarly when there are five or more reports of a particular product-event pair (CIOMS, 2010, pp. 59) • Bayesian methods may provide a solution to “false positive” indications in large datasets • It is important to note that the literature does not demonstrate consensus on cost/utility of various data mining tools with respect to specificity and sensitivity – (CIOMS, 2010, pp. 61) • As such, company specific decisions must be discussed as part of the signal management processIntroduction to Signal Detection 29
  30. 30. Technical Solutions… • Use of these databases requires that certain assumptions be made – Drugs used in the marketplace are used by a representative sample of the greater population • Any information derived from these databases should be interpreted using the limitations of the data contained therein (Edwards, 1999) – Limited clinical quality of data • USA allows reporting into the AERS system from anyone (Health care provider {HCP} or not) • EMEA only allows reporting by HCP thus typically more complete clinical informationIntroduction to Signal Detection 30
  31. 31. Technical Solutions… • Three main products are available for large data mining opportunities – Qscan by Drug Logic – Empirica Signal by Oracle – agSignals by Aris Global – dsExplorer by Cerner • In the spirit of full-disclosure, BioPharm Systems is a Gold Partner with Oracle and the next slides are taken from the Empirica Signal product – Please note that except for very specific functional differences, these software systems are designed to accomplish similar tasks. The degree to which one is better than the other is not discussed in this presentation and the use of slides is not an endorsement of one product over another – As already stated, the decision to evaluate signals and the method used is a company specific decisionIntroduction to Signal Detection 31
  32. 32. Technical Solutions… • Empirica Signal can display disproportionality results in a sector map fashion which allows for visual assessment of signal strength.Introduction to Signal Detection 32
  33. 33. Technical Solutions… • In this example of “Ziconotide”, we see that sector 2 is “Vertigo” with a disproportionality ratio of 2.052 – This is a good example though of a well know substantiated concern with the product as it is known to disturb motor function and balanceIntroduction to Signal Detection 33
  34. 34. Technical Solutions… • Sector 1 is “Tinnitus” with a disproportionality ratio of 4.194Introduction to Signal Detection 34
  35. 35. Technical Solutions… • The software solutions available provide a wonderful way to quickly and comprehensively analyze marketed data reported to regulatory agencies. • These data are subject to the issues surrounding the collection of the information in them – Underreporting of serious events • Changes the number of expected events • “Weber Effect”: The peak reporting for events in a drug on market occurs within the first 2 years of approval (Hartnell, & Wilson, 2004) during the initial 5 year marketing period – Over reporting of events of non-interest (expected non-serious)Introduction to Signal Detection 35
  36. 36. Technical Solutions… • False Causality attribution – Signals ARE NOT CAUSAL INDICATIONS – They are disproportionate reporting indicators • Mitigation of these limitations can occur if you establish a signal prioritization method in your company for dealing with the various signals identified by manual or automated signal detectionIntroduction to Signal Detection 36
  37. 37. Signal Prioritization… Triage of Outputs · Interpret the signal in context of other relevant sources, disease knowledge, biologic plausibility, alternative etiologies, etc. Monitor If signal is indeterminate NO Impact Is signal assessment and Need further NO YES investigation refuted? prioritization Close out YE S (if signal is refuted)Introduction to Signal Detection 37
  38. 38. Signal Prioritization • Prioritization of signals is still a very controversial aspect of signal management (Waller, 2010 p. 50) • Two proposed methods are as follows: – WHO – Triage (CIOMS, 2010 p. 88; Lindquist, 2007) – MHRA – Impact analysis (draft literature)Introduction to Signal Detection 38
  39. 39. WHO - Triage • Seriousness assessment – Is the event serious or not? • Unexpectedness – Is the event expected or not? • Disproportionality score – Is the score high or not? • Temporal displacement of score – Has the disproportionality score increased over time?Introduction to Signal Detection 39
  40. 40. WHO - Triage • Temporal occurrence – Is the event occurring within the first few years of launch • Careful as Weber effect could contribute to confounding here • Multiple signaling countries – Are more than one country seeing this issue? • Positive Rechallange – Is there evidence of positive rechallange? • Specialty list of terms of interest – Is this a term of interest as identified by the companyIntroduction to Signal Detection 40
  41. 41. MHRA – Impact analysis… • A calculated quantitative score based on “Evidence” and “Public Health” – Evidence Score • Degree of disproportionality (PRR, IC etc) • Strength of evidence • Biologic Plausibility – Public Health Score • # of reported cases per year • Expected health consequences • Reporting rate in relationship to the level of drug exposureIntroduction to Signal Detection 41
  42. 42. MHRA – Impact analysis • Results in the following quantitative categories: – High – Need to gather more information – Low – No action • Additionally the MHRA is looking at a method to incorporate the following in a prioritization scheme: – High profile product (media attention) – Risk perception by general population – Political obligationsIntroduction to Signal Detection 42
  43. 43. Signal Prioritization • Regardless of the method used, every signal should undergo this type prioritization – CIOMS (2010 p. 22) suggest that this effort results in the following outcomes which I have labeled in the following manner: • Confirmed signal – Results in the movement of the signal to the evaluation process • Unconfirmed signal – Results in the monitoring of this event over time and regularly re-prioritizing it based on new information • Refuted signal – Results in the closing of the signal » Disease progressions » Known issues etc.Introduction to Signal Detection 43
  44. 44. Signal Prioritization • Regardless of the method used, care should be taken in that this type of assessment and analysis should be an iterative process when new information is amassedIntroduction to Signal Detection 44
  45. 45. Exercise 4 1. Please segregate into your small group as assigned by the index card in your chair 2. Using collaborative discussion and the information provided, Signal please prioritize the various Prioritization signals using one of the above methods (WHO or MHRA) 3. Nominate a single spokesperson from the group to share the findings with the larger audience.Introduction to Signal Detection 45
  46. 46. Signal Evaluation… Signal Evaluation · Individual case review · Aggregate analysis · Periodic reports · Non-interventional studies · Non-clinical studies · Class analysis · Other relevant informationIntroduction to Signal Detection 46
  47. 47. Key Components to Signal Evaluation… • Once a signal is prioritized as “Confirmed” or its equivalent, further data must be gathered to further the evaluation of the signal (Waller, 2010 p. 50, 51) • Causality – Does the provided evidence support a causal relationship between the product and event? • ICSR Causal Relationships – Probable, possible, unlikely, unrelated, unassessable Causality is more than thisIntroduction to Signal Detection 47
  48. 48. Causality… • Bradford-Hill (Shakir & Layton, 2002) proposed the following categories should be used to assess causal relationships found in data – Strength • The stronger an association, the less likely it is explained by other factors – Consistency • Multiple sources demonstrate similar associations – Temporality • Is the event consistently treatment emergent? – Positive re or de challenge evidenceIntroduction to Signal Detection 48
  49. 49. Causality… • Bradford-Hill (cont.) – Biologic Gradient • Evidence of dose or duration related risk – Dose-dependency or cumulative exposure over time – Specificity • Single product exposure results in event – Presence of this leaves little doubt to causality – A few adverse drug reactions in an of themselves are syndromes • Absence of this does not reflect a non-causal relationship – For example nicotine exposure and lung cancersIntroduction to Signal Detection 49
  50. 50. Causality… • Bradford-Hill (cont.) – Plausibility • Does the existing literature support this association? – Coherence • Is the association compatible with the existing literature and knowledge? – Experimental Evidence • Is there data that demonstrates the event can be altered or eliminated by some experimental regimen? – Converging evidence between post marketed and clinical surveillance – Analogy • Are there existing alternative explanations for the association in the literature? • Absence of these strengthens the causal likelihood • Absence of any or all of these four criteria does not indicate the absence of a causal relationship since out data may be describing a newly seen observation not part of the literature used in these assessmentsIntroduction to Signal Detection 50
  51. 51. Causality… • In general, the more of the criteria satisfied, the stronger the causal relationship. The assessment of how many and which criteria are more important than others, is no simple formula and judgment is required (Waller, 2010 p. 29) • There are issues with the PV databases and literature used in signal evaluation and one must keep these in mind when applying the Bradford-Hill criteria (Shakir & Layton, 2002)Introduction to Signal Detection 51
  52. 52. Key Components to Signal Evaluation… • In addition to assessments of causality, the concept of Frequency is important in signal evaluation (Waller, 2010 p. 51) – The question of frequency can be categorized into the following measures of prevalence • Very Common – More than 1:10 • Common – 1:10 to 1:100 • Uncommon – 1:100 to 1:1000 • Rare – 1:1000 to 1:10,000 • Very Rare – Less than 1:10,000Introduction to Signal Detection 52
  53. 53. Key Components to Signal Evaluation… • In addition to assessments of causality and frequency, the concept of Clinical implications is important in signal evaluation (Waller, 2010 p. 51) – Clinical Implications can be rephrased to impact on patient health • Is the event life-threatening? • Does its presence lead to a congenital anomaly? • Does its presence lead to death? • Does its presence lead to long term disability or long term hospitalization?Introduction to Signal Detection 53
  54. 54. Key Components to Signal Evaluation… • In addition to assessments of causality, frequency and clinical implications, the concept of Preventability is important in signal evaluation (Waller, 2010 p. 51) – Preventability • If an intervention could be applied at this stage, would the event of interest or its outcomes be prevented?Introduction to Signal Detection 54
  55. 55. Key Components to Signal Evaluation… • To summarize, every confirmed signal is evaluated in terms of the following: – Causality – Frequency – Clinical implications – Preventability • Based on the suspected signal’s evaluation outcome, you may choose to elevate it to the status of a potential or identified risk (CIOMS, 2010 p. 93, 94)Introduction to Signal Detection 55
  56. 56. Key Components to Signal Evaluation • This signal evaluation process is intimately tied to risk identification and may result into the feeding of the signal into the risk mitigation/management planning processes at your company. • The information collected in the signal evaluation phase will feed the safety specification section of the risk management plan.Introduction to Signal Detection 56
  57. 57. Exercise 5 1. Please segregate into your small group as assigned by the index card in your chair 2. Using the provided clinical information, is there evidence Signal Evaluation that “Ascites” should be elevated to a risk? 3. Nominate a single spokesperson from the group to share the findings with the larger audience.Introduction to Signal Detection 57
  58. 58. Summary 1. Create a procedure to formalize signal management in your organization • Ensure your process includes signal detection, prioritization and evaluation 2. Signal detection efforts should include both qualitative and quantitative methods Top Five 3. Signal prioritization should describe the objective methods used to categorize the signals Take- discovered through detection efforts Aways 4. Signal evaluation should be used to document and formally evaluate only those critical signals (from prioritization methods) 5. Information collected from your formal evaluation strategies can be used to seamlessly parse into your risk management processIntroduction to Signal Detection 58
  59. 59. References • Council for International Organizations of Medical Sciences (CIOMS). (2010). Practical Aspects of Signal Detection in Pharmacovigilance. Report of CIOMS Working Group VIII, Geneva . • Egberts, A.C.G., Meyboom, R.H.B., and van Puijenbroek, E.P. (2002). Use of Measures of Disproportionality in Pharmacovigilance: Three Dutch Examples Drug Safety 25(6): 453-458 • Gajewski, B.J. and Simon, S.D. (2008). A One-Hour Training Seminar on Bayesian Statistics for Nursing Graduate Students. The American Statistician, 62 (3) • Hartnell, N.R. and Wilson, J.P.. (2004). Replication of the Weber Effect Using Postmarketing Adverse Event Reports Voluntarily Submitted to the United States Food and Drug Administration. Pharmacotherapy. 24:743- 749 • Hopstadius, J., Norény, G.N., Bate, A., and Edwards, R.. (2008). Impact of Stratification on Adverse Drug Reaction Surveillance. Drug Safety. 31(11) • International Conference on Harmonisation (ICH). (2003). ICH Harmonised Tripartite Guideline. Post-approval safety data management: Definitions and standards for expedited reporting E2D. Retrieved on July 26. 2011 from • Levitan, B., Yee, C. L., Russo, L., Bayney, R., Thomas, A. P., and Klincewicz, S. L. (2008). A Model for Decision Support in Signal Triage. Drug Safety, 31 (9) • Lindquist, M. (2007). Use of triage strategies in the WHO signal-detection process. Drug Safety, 30:635-7 • Maggid. (2011). Retrieved from on August 31, 2011 • Shakir, A.W., and Layton, D.. (2002). Causal Association in Pharmacovigilance and Pharmacoepidemiology: Thoughts on the Application of the Austin Bradford-Hill Criteria. Drug Safety. 25 (6): 467-471 • Strom, B. L., and Kimmel, S. E. (2006). Textbook of Pharmacoepidemiology. Wiley, West Sussex, UK • Waller, P. (2010). An Introduction to Pharmacovigilance. Wiley-Blackwell. Oxford, UKIntroduction to Signal Detection 59
  60. 60. Contact Information Rodney has over 15 years experience in clinical research including laboratory experimentation, clinical data management, clinical trial design, dictionary coding and pharmacovigilance. Rodney has worked for BioPharm Systems for eleven years now serving in a variety of roles all related to the technical and/or clinical implementations of software systems used in the clinical trial process. Prior to coming to BioPharm Systems Rodney worked at pharmaceutical and technology companies in the Dictionary Coding, Statistical Programming and Data Management areas. In addition to his current work at BioPharm Systems, Rodney holds an Associate faculty position at Walden University teaching Public Health Informatics and other health information systems courses. Rodney holds a Bachelor of Science in Genetic Engineering, a Masters of Public Health in International Epidemiology and a Ph.D. in Epidemiology focusing on Social EpidemiologyIntroduction to Signal Detection 60