This document summarizes a sensitivity analysis approach to "observed versus expected" analyses that accounts for uncertainties around reporting bias and background incidence rates. It presents a visual framework called an "OE-plane" that plots the observed number of reports against the expected number for a range of background incidence rates and reported fractions. This allows evaluating whether an excess of observed reports could be due to bias or is robust to plausible variations in the assumptions. The approach provides regulators with information to draw their own conclusions based on which ranges of rates and fractions they deem most relevant.

1. ICPE 2014, Taipei, Taiwan, October 24–27, 2014
Beyond traditional
“Observed versus Expected” analyses
A sensitivity analysis integrating uncertainties
around reporting bias and background
incidence rate
Lionel Van Holle, Olivia Mahaux, Vincent Bauchau
Vaccine Clinical Safety and Pharmacovigilance, GSK Vaccines, Wavre, Belgium

2. Disclosures and Acknowledgements
• The project was funded in its entirety by GlaxoSmithKline Biologicals
SA
• The following personal or financial relationships relevant to this
presentation existed during the past 12 months:
– Employment by and owning restricted shares from GSK group of
companies
• Editorial and publication coordination support was provided by
Véronique Delpire and Mandy Payne (Words & Science) and XPE
Congress Team (XPE Pharma & Science) both for GlaxoSmithKline
Biologicals SA.

3. What are “Observed vs Expected” analyses?
• Quantitative analysis for signal strengthening
When is it used?
• After signal detection and before formal pharmaco-
epidemiological study, if any
• Generally, for short-term, acute events
• Usually for vaccines
Why is it used?
• Produced quickly as relying on spontaneous report data
and available background incidence rates
Background

4. The Observed
• A condition for which a signal has been generated: has
to be medically defined using a list of medical terms
• A suspected risk period post vaccination based on
independent data and biological plausibility
• An observed number of spontaneous reports within
the suspected risk period being confirmed (different levels
of certainty) as having the medical condition (regardless
of causality assessment)
MedDRA PTs, Medical Dictionary for Regulatory Activities Preferred Terms

5. The Expected
[Expected within the Risk Period] =
[Background IR] * [Person_time at risk]
Under null hypothesis of no causal association of the
condition to the vaccination
IR, incidence rate

6. [Expected within the Risk Period] = [Background
IR] * [Person_time at risk]
Background IR
• Should come from a population as close as possible to
the vaccinated population (demographic characteristics,
calendar years, region, etc.)
• The definition of the condition should be consistent with
the definition used for the observed
Person_time at risk
• Person_time at risk = exposed persons * suspected risk
period
 Sales data are often used as a proxy of the exposure

7. Limitations
• Very important uncertainties such as:
– Relevance of background incidence rates
– Reporting bias
– Diagnosis
–Real use of sold doses
• Usual Sensitivity approaches may seem to be objective
but they usually cover only a limited number of scenarios

8. • Rosenbaum’s definition of bias analysis:
« Ask how much hidden bias would need to be present
to alter the study’s qualitative conclusion »
Rosenbaum PR. Journal of Educational Statistics 1989
• We applied the Rosenbaum’s concept
• We presented the results of the Observed vs Expected
(OE) analysis visually in an OE-plane
Where x-axis and y-axis represent the parameters associated with
the highest uncertainty in the calculation of the expected.
Methods

9. Results
• For illustration, we considered these two parameters as
the main source of uncertainty (often the case for rare
conditions):
– The background IR
–The reported fraction
• We developed a visual framework to answer the question
about a potential excess of reported cases compared to
what is expected and account for main uncertainties.

10. Results – OE analysis
IR, incidence rate; py, person-years

11. Conclusion
• Depending on how plausible the range of background
incidence & reported fraction is, a conclusion regarding a
potential excess of observed cases vs the expected can
be drawn.
• The framework also allows regulatory authorities to draw
their own conclusion should they find another range of
background incidence rates & reported fraction more
relevant