This document discusses quantitative methods for signal detection in pharmacovigilance. It describes current methods like proportional reporting ratios (PRR) that measure disproportionate reporting of adverse drug reactions. Bayesian methods and log-linear models are presented as improvements. Longitudinal health records are discussed as an alternative that avoids some biases but is difficult to implement. Future directions may include using all available data on reports and incorporating pharmacokinetic properties of drugs.

1. An agency of the European Union
Presented by: Name Surname
Position/organisational descriptor
Quantitative methods of
signal detection
François MAIGNEN

2. 2
Learning objectives
•Current methods: Disproportionality analysis
•Improvement of the current methods: Bayesian
methods and (Bayesian) log-linear models
•Comparison of the methods (limitations, stats vs
clinical)
•Star wars – a new hope? Longitudinal health
records: better? Different and not devoid of
biases
•Future directions

3. 3
Measures of disproportionate reporting
Most of the methods routinely used in
pharmacovigilance (spontaneous reporting
systems) databases are based on measures of
disproportionate reporting (i.e. ROR, PRR, BCPNN,
MGPS, etc …).
Basically: “Observed vs Expected” analysis in a
given database i.e. % of reports involving a given
reaction for a given medicine compared to the %
of reports involving this reaction on the whole
database

4. 4
A spontaneous reporting system database
SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N
Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N
Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N
Event 3 n31 n32 … … … … … … n3N
Event 4 n41 n42 … … … … … … n4N
Event 5 n51 n52 … … … … … … n5N
Event 6 n61 n62 … … … … … … n6N
… … … … … … … … … …
Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN

5. 5
Drug 1 All other
medicinal
products
Total
Event 1 a c
All other
reaction
terms
b d
Total N = a +
b + c + d
c + d
a + c
a + b
Proportional Reporting Ratio

6. 6
Proportional Reporting Ratio
PRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN
IN PRACTICAL TERMS?
a/(a+b) = Proportion of reports involving a
specific adverse event among all the reports
involving DRUG A
c/(c+d) = Proportion of reports involving THE
SAME adverse event among all the reports of your
database but DRUG A

7. 7
Proportional Reporting Ratio
If the rate of reporting of AE for drug 1 is similar
to the rate of reporting of this AE for all the other
products of the database, the PRR will be equal to
1 (same proportion of reports involving the
reaction for drug A than for the other drugs) …
BUT … If the reaction is proportionately MORE
reported with drug A than for the other products,
the PRR will be increased (typically > 1).
 DIS-PROPORTIONALITY of reporting

8. 8
Disproportionality analysis (example)
. CNS drug for which the total No of reports is
400, of these 20 reports of diarrhoea
. All other products in the database (1 million
reports excluding reports involving drug A), of
these 50,000 reports of diarrhoea.
PRR = [20/400] / [50,000/1,000,000] = 1 (no
SDR)

9. 9
Disproportionality analysis (example)
CNS drug for which the total No of reports is 400,
of these 40 reports of drowsiness
. All other products in the database (1 million
reports excluding reports involving drug A), of
these 25,000 reports of diarrhoea.
PRR = [40/400] / [25,000/1,000,000] = 4
(presence of a SDR)

10. 10
Improvements of these methods
•Considering possible confounding factors:
stratification and log-linear models (ROR – see work
from E. Van Puijenbroek)
•Trying to circumvent low expected values or low case
counts: Bayesian models (A. Bate & W. DuMouchel)
•Other regression methods: LASSO and Bayesian
logistic regressions (N. Noren, D. Madigan)
•Clinical relevance not always clear or demonstrated
•Some methods can be computationally demanding

11. 11
Bayesian methods
BCPNN and MGPS rely on the same principle of
conjugate prior distributions:
•These methods will shrink the value of the
measure of disproportionality using a Bayesian
approach (prior based on existing dataset)
•BCPNN: cell counts ~ Binomial dist., conjugate
prior = beta
•MGPS: cell counts ~ Poisson, conjugate prior =
Gamma (mixture of Gammas).

12. 12
Bayesian methods
Assume binomial y=7
successes, 20 trials.
Non informative prior =
Beta (2,4)

13. 13
Thresholds - ARBITRARY
All these methods provide a ranking …
Thresholds = arbitrary
Trade-off between
•Reviewing too many drug-event pairs
(loss of operational benefit)
•Missing some signals
No ADR ADR

14. 14
Comparison of the methods
Methodological difficulties
No gold standard / no standardised reference
method
Imprecision of what constitutes a signal
Retrospective vs prospective evaluation
Importance of clinical judgement. The added
value of clinical evaluation is currently unknown
(if any).

15. 15
Comparison of methods
1
2
3
4
7
11
6
9
10
12
5
8
1
2
3
4
7
11
6
9
10
12
5
8
4
2
3
1
7
11
6
9
10
12
5
8
Meth.1
Meth.2
Meth.3
• Threshold 1: Meth. 2 = 5
true signals, meth. 1/3 =
4.
• Threshold 1+2: Meth.
2=Meth.3
• First 5 signals: Meth. 1
≠ Meth. 2 = 3.
ADR No ADR

16. 16
Comparison of methods (PRR vs BCPNN)

17. 17
Performances of these methods
Operational benefit (screening of large databases)
Anecdotal evidence (in opposition to structured)
of signals discovered thanks to the quantitative
methods (recent examples incl. D:A:D and MI)
Time benefit in some cases (Hochberg & EV
study)
NND ~ 7/15 (depending whether the study is
retrospective or prospective)
Idea: Quant. Methods + DMEs/TMEs

18. 18
New approaches to signal detection
Deviation of Obs. vs Expect. distr. from a fitted
distribution (Jim)
Modelling of the hazard function of the time to
onset (DSRU / François) hazard # mechanism
Use of longitudinal databases (record linkage and
electronic health records – OMOP / Noren /
Callreus) ~ incidence rate ratio
• Same patients different time windows (A. Bate)
• Hospital records of different patients (T. Callreus)

19. 19
Merci Andrew

20. 20

21. 21
Interpretation / Limitations
Patients prescribed a PPI are associated with
acute pancreatitis in the month after the
Prescription (ICdiff positive)
But graph shows that these patients are general
likely to have acute pancreatitis around the time
of PPI prescribing (In agreement with confounding
by indication)

22. 22
Record linkage

23. 23

24. 24
Interpretation / limitations
Bias associated with the hospitalisation
(confirmed by later events which occurred
remotely after the administration of the vaccine)
Spurious / unexplained associations
Rely heavily on temporal association (Post hoc
ergo propter hoc)

25. 25
Longitudinal health records
Powerful to detect some associations
Even if reporting artefacts do not influence this
method, not devoid of other biases (selection,
protopathic, misclassification, etc …)
Much more complicated to implement (very large
datasets, confidentiality aspects, linkage of
records and interoperability of databases, etc …)

26. 26
Future directions (?)
•Performances in a prospective setting, added
value of clinical judgement needs to be quantified
•Duplicate detection
•Other approaches to signal detection (use all the
information included in the reports, PK/PD
properties of the substances)
•Surveillance networks (incl. spontaneous
reporting)

27. 27
Key elements
•PRR is a measure of disproportionality of
reporting in a specific database (observed vs
expected value computed on the whole database)
•A SDR indicates a possible signal which needs to
be medically confirmed
•All other methods proceed from the same logic
•Data mining on EHR will probably improve the
detection of signals (organised method of
collection) but is not free of biases and is
resources demanding

28. 28
Bibliography
Guideline on the use of statistical signal detection methods in EVDAS
http://www.ema.europa.eu/pdfs/human/phvwp/10646406enfin.pdf
Bate & Evans quantitaitve signal detection using ADR reporting. PDS
2009.
Hauben & Bate. Decision support methods for the detection of adverse
events in post-marketing data. Drug Discovery Today. April 2009.
Evaluation of Signal Detection Methods for use in Prospective Post
Licensure Medical Product Safety Surveillance [FDA Sentinel Initiative
Safety Signal Identification Contract] – Report
http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm149343.htm