1. Matching key safety questions with
appropriate algorithms for
appropriate corrective actions
Lionel Van Holle (GSK Vaccines)
DSRU 10th June 2015

2. Table of content
Introduction
The key Safety questions
The appropriate corrective actions
The role of algorithms screening SRDs
Disproportionality
Time-to-onset signal detection
Logistic regression
Tree-based scan statistics
The full quantitative signal detection toolkit
Future developments needed
Conclusion

3. Introduction
• Disproportionality algorithms have been used for
screening spontaneous report databases (SRDs)
for more than a decade.
• Are they enough or should we develop other
algorithms?
• What for?
– Better answering the same safety question as before?
– Answering other safety question?

4. The Key Safety Questions
1. Does the product cause adverse reactions?
Product-related safety issues
2. Does an ingredient of my product cause adverse
reactions?
Ingredient-related safety issues
3. Does a subset of manufactured products cause
adverse reactions?
Manufacturing-related safety issues

5. Safety issues Examples
Product-related Thalidomide drug for morning sickness -> birth
defects, limb malformations in ~ 10,000 children
worlwide
Ingredient-related E-ferol: an injectable preparation of alpha-tocopherol
(vitamin E) for parenteral nutrition was recalled from
the market because of unusual liver and kidney
syndromes with 38 deaths reported among treated
low birthweight infants -> syndrome most likely due
to a combination of alpha-tocopherol, polysorbates,
contaminant.
Manufacturing-related Cutter incident -> some lots of the Cutter vaccine
(polio) were not properly inactivated and contained
live polio vaccine -> 120,000 doses distributed;
40,000 developed abortive poliomyelitis; 56
paralytic; 5 deaths

6. The appropriate corrective actions
• ? Safety profile re-evaluation
• ? B/R re-evaluation
Product-related
safety issue
• ? Strategy of ingredient
substitution/removal
Ingredient-related
safety issue
• ? Recall of manufacturing lot(s)
• ? Development of new QC/QA
tests
Manufacturing-
related safety issue

7. The role of algorithms screening SRDs
Due to high number of spontaneous reports
preventing individual medical assessment
All product-event pairs
First-pass
screening
Causality
Assessment
Algorithms that do not require
prior medical assessment
Association
Temporality
Specificity
Consistency
Biological gradient
Experimentation
Plausibility
Analogy
SAFETY SIGNALS

8. Disproportionality
When disproportionality is used in routine, it
compares the observed number of reports for a
given product-event to what is expected from
other/all products.
Event of interest Other (or all) events
Product of interest A B
Other (or all) products C D
‫ܣܲܦ‬ ൌ
ಲ
ಳ
಴
ವ
ൌ
஺ ∗ ஽
஻ ∗ ஼
ൌ
஺
ಳ ∗಴
ವ
ൌ
ை௕௦
ா௫௣

9. Time-to-onset signal detection
Kolmogorov-Smirnov tests
If the distance between
Cumulative distributions is
unexpected
Van Holle L et al, Using time-to-onset for detecting safety signals in spontaneous reports of adverse events
following immunization: a proof of concept study. PDS 2012; 21: 603-610. DOI: 10.1002/pds.3226

10. Logistic regression
• Determines if the reporting pattern (in terms
of causality criteria/strength of evidence) of a
product-event pair is similar or not to the
reporting pattern of a positive reference set
(i.e emerging signals or listed events)
Van Holle L et al, Use of logistic regression to combine two causality criteria for signal detection in vaccine spontaneous
report data. Drug Safety (2014) 37:1047-1057.
Caster O et al, Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence
aspects in vigiRank. Drug safety (2014) 37: 617-628.

11. • Integrate more causality criteria in the first-pass
screening
BUT
• Routine signal detection methods (DPA or more
advanced ones: TTO, LogReg, Supervised methods) use
other products as comparator and are consequently
inappropriate for detecting ingredient-based or
manufacturing-based safety issues.
• It leads to potential non-detection of these issues or
worse, detection at the wrong level (product).
Role of these methods?

12. Relevant Action performed ?
Safety profile update
•LIKELY if product related
•POSSIBLE if large fraction of lots
Ingredient substitution
•UNLIKELY if ingredient shared
(Rely only on qualitative
assessment or ad hoc analysis)
Lot(s) recall
•UNLIKELY if small fraction of lots
(Rely only on qualitative
assessment or ad hoc analysis)
Algorithms
Routine Disproportionality [+ TTO, LogReg]
Safety issue
Product-related Ingredient-related Manufacturing-related

13. Tree-based scan statistic
Originally used in disease surveillance: e.g.
investigating ‘death from silicosis’ (event of
interest) incidence among different occupations
or group of occupations (exposure).
Potential cuts in the tree structure
symbolizing a scanning window
investigating combinations of
occupations

14. • For each tree cut, the rate of the event of interest (λ)
in the window scan (G) or not (R) is calculated. Total
number of cases of interest if fixed (c).
• Null hypothesis (H0) is that the rate of the event-of-
interest is the same in the window scan than outside.
• Alternative hypothesis (H1) is that the rate of the
event-of-interest is higher in the window scan than
outside.
• A likelihood ratio can be built (H1/H0)
Kulldorf M et al, A tree-based scan statistic for database disease surveillance. Biometrics (2003) 59, 323-331.

15. • The cut with the maximal likelihood ratio constitutes
the test statistic
• Significance is measured through Monte Carlo
simulations
• It adjusts for multiple testing across the multiple cuts
in the tree structure.
It can be adapted from disease surveillance to adverse
reaction surveillance if we link spontaneous report data
with hierarchical data representing an exposure of
safety relevance.
Kulldorf M et al, A tree-based scan statistic for database disease surveillance. Biometrics (2003) 59, 323-331.

16. Has the potential for being an algorithm for detecting
manufacturing-related issues & identifying most likely
manufacturing step
Specific
to SRDs

17. Has the potential for being an algorithm for detecting
ingredient-related issues & identifying most likely
ingredient

18. Relevant Action performed ?
Safety profile update
LIKELY if product related
Ingredient substitution
LIKELY if ingredient-
related
Lot(s) recall
LIKELY if manufacturing-
related
Algorithms
Routine Disproportionality
[+ TTO, LogReg]
Tree-based scan (linked to
product dictionary)
Tree-based scan (linked to
manufacturing data)
Safety issue
Product-related Ingredient-related Manufacturing-related
Full quantitative SD toolkit

19. Future developments needed?
• Spontaneous report database (SRD) need to
be linked to external information:
– A product dictionary with a hierarchical structure
allowing to see the different ingredients of the
product (non-independence of the products)
– A manufacturing database with a hierarchical
structure allowing to see the different
manufacturing steps (non-uniformity in
production)
>< DPA approach with standalone SRD, flexible
enough softwares?

20. • Define which events to monitor for
manufacturing-related or ingredient-related
safety issues:
– All MedDRA PTs as for product-related safety? (no a
priori)
– A subselection based on biological plausibility?
– A grouping of terms?
– …
• Extend the scope of tree-based scan statistic to
allow integration of other causality criteria (than
numbers) as for product-related safety issues?
• Need a zero-pass screening to determine the
most likely scenario (product, manufacturing,
ingredient)?

21. Conclusion
• Quantitative signal detection toolkit should
contain algorithms of signal detection able to
detect different types of safety issues (product,
ingredient, manufacturing)
• The tree-based scan statistic is a good candidate
for filling the current gap that prevents
appropriate corrective actions
• Creation of a product dictionary & a
manufacturing hierarchy database will be
required.