My talk on pharmacovigilance and social media - and why we're overlooking the most important factors that determine results.

1. A Real-World Look at Mining Social Media for
Adverse Events: Impact of Regulatory Definitions
and Methods
Michael A. Ibara, Pharm.D.
Head of Digital Healthcare
CDISC*
*(Formerly Pfizer, Inc., during which time research was conducted.)
@michaelibara

2. A note on the distribution version of these slides:
These slides are designed specifically to support the talk I gave, and not as a
standalone document. They provide visual support during my talk, and I make
many points verbally that are not spelled out in the slides. I apologize, then, if
some points seem opaque because of this, and if you want to know more I’m
happy to have a quick chat or email exchange to explain anything.
Having said this, I hope you can glean some information from them about the
rationale and direction in this talk. I welcome any discussions!
You can reach me at mibara@cdisc.org, or at michael.ibara@gmail.com
Thank you!

3. The ideas and opinions in this talk are
mine and not necessarily those of
CDISC, my colleagues, or indeed any
other human being

4. Tufts CSDD Social Media and Drug Development Study
General
Practices
Guidelines
for company-
owned sites
Guidelines
on
resourcing
Program
Planning
Best
practices on
open
innovation
Best
practices on
crowdsourc-
ing
Best
practices on
patient
reported
outcomes
Patient
Recruitment
Effective
global
recruitment
strategies
Recomme-
ndations for
future
recruitment
Pharmaco-
vigilance
Recomme-
nded policy
guidance
Principles for
identified AE
scenarios
Social Listening
Strategies
behind
listening
effectively
Operation-
alizing social
listening

5. M. Ibara1
, S. Stergiopoulos2
, J. Van Stekelenborg3
, A.C. Ianos4
, R. Ruben5
, P.N. Naik6
, R. Boland7
.
1
CDISC previously Pfizer- Inc, Pharmacovigilance, New York, USA
2
Tufts Center for the Study of Drug Development, Project Management, Boston, USA
3
Johnson & Johnsn, Lead Methods and Analysis, New York, USA
4
Pfizer- Inc., Safety Risk Management, London, United Kingdom
5
Independent formerly ParagonRx International- LLC, Risk Management, Philadelphia, USA
6
Independent formerly Tufts Center for the Study of Drug Development, Research Analyst, New
York, USA
7
Janssen- Pharmaceutical Companies of Johnson and Johnson, Translational Informatics &
External Innovation R&D IT, Philadelphia, USA
Colleagues

6. The premise...
6
Growing sophistication in
mining social media for
possible reports of AEs

7. PV INTEREST IN SOCIAL MEDIA
A personal timeline
c. 2004
c. 2006
What?
I’ll wait!

8. c. 2010
Tell me
more!
Don’t
tell me!
I’m
scared!
c. 2013
Ok, let’s
figure this
outThis might
be good!

9. c. 2014-15
c. Now
We’re getting
the hang of
this
Piece of
cake!

10. The Premise...
10
Growing interest in
determining how such
activities are interpreted in
light of regulations
Growing sophistication in
mining social media for
possible reports of AEs
Academic studies address
utility of mining social media
for AEs, but real-world
scenarios match up poorly
with ideal research
conditions
It is not always clear to
what the extent actual
operational designs
affect the results of that
work

11. Objectives
Determine the real-world ability to obtain reproducible results
using a single definition of “reporter” to mine social media for
possible AEs
Using available vendors
Allowing standard approaches within limits
Test whether the operational definition of “reporter” has a
direct impact on results (i.e., counts of possible AEs)
Determine the extent to which varying the definition of
“reporter” changes the counts of possible AEs
11

12. Design: 6 Vendors / 9 Drugs
• 6 vendors recruited based on their stated capabilities of mining social media for
possible AEs
• Predetermine set of drugs was used for the investigation* None were marketed by
companies whose members participated in the investigation:
• Olanzapine (Lanzek, Zypadhera, Zyprexa, Symbyax)
• Trazodone (Depyrel, Desyrel, Molipaxin, Oleptro, Trazodil, Trazorel, Trialodine, Trittico)
• Lamotrigine (Lacmictal)
• Natalizumab (Tysabri, Antegren)
• Aripiprazole (Abilify, Aripiprex)
• Esomeprzole (Nexium, Essocam)
• Duloxetine (Cymbalta, Ariclaim, Xeristar, Yentreve, duzela, Dulane)
• Nicotine (Nicotrol, Habitrol, Nicoderm, Nicorette, Nicotinell, Commit, Thrive)
• Aspirin
*Comparable with Leaman, Robert, et al.2010; Nikfarjam, Azadeh, et al. 2011
12

13. Design: Retrospective Mining
• Retrospective mining of social media for possible AEs for each listed drug
• From 1Jan2013 to ‘present’ to reach at least 400 hits per drug
• Greater time period can be investigated if needed to meet hits requirement
• Possible AE is defined as containing the “4 elements”
• Identifiable event, patient, drug, reporter
• Three definitions were standard
• “Event” as defined broadly in regulations and defined operationally per vendor (multiple symptoms in
a single post are defined as a single ‘hit’)
• “Patient” defined broadly in regulations and in absence of vendor’s definition will be “Knowledge of an
individual experiencing the event with at least one of the following patient qualifiers: a pronoun or
noun implying a human; an age or age category; gender; initials; date of birth; name; or patient ID
number
• “Drug” as the drug in question using vendor’s procedures.
13

14. Identifiable Reporter – 4 Levels
• LENIENT
• Post exists, i.e., no requirement to identify a reporter
• Used in the initial data collection
• LOW
• Any type of information suggesting there is an actual reporter (e.g., acronym, pseudonym, proper
name, email address)
• STANDARD
• At least one piece of identifiable information for the reporter - i.e,. local identifier for the platform
that allows contacting (e.g., Facebook name) OR a validly formatted email address OR validly
formatted phone number
• STRICT
• Must match standard criteria and in addition have additional piece of identifying information such
as valid phone number, valid user name from site, mention of geographic location of the person in
question
14

15. “Hits”
• For each drug a minimum of 400 ‘hits’ according to the Lenient definition was
the goal
• Type of reporter identifying information was collected by vendors, but was
masked to subteam investigators and supplied as counts
• E.g., 10 cases with single valid mail addresses, 5 cases with both valid email
addresses and user names
• Search was limited to English
15

16. Design: Data Sources / Collection
• Four standard data sources
• Facebook
• Twitter
• Dailystrength.org
• Drugs.com
• Vendors varied in terms of methods of collection and extent to which sources
were utilized
• As this was a real-world examination, no attempt was made to develop a single
consistent dataset
• No data was collected that was not publically available
16

17. RESULTS

18. Results: Real-world Variability
•Began with 6 vendors claiming ability to independently
complete request. Completed with 4 vendors
•Significant differences across vendors in methods
• Operational definitions of drugs, terms
• Lexicon use, sophistication
• Sourcing and amount of data collected
• Algorithms used
• Type and extent of curation
18

19. Results: Overall Findings
• It is not possible to pool results or make direct comparisons given
the variability in methods
• Each set of results must be treated individually based on the
unique set of methods used
19
Vendor Facebook (‘hits’ per examined) Twitter (‘hits’ per examined)
1 0 / 34 17 / 239
2 43 / 11,431 [human] 141 / 37,409 [human]
3 373 / 9,823 [human] 32 / 33,167 [human]
4 1,200 / 85,220 1,318 / 105,018
Example: ‘Hits’ on Facebook and Twitter by Vendor

20. “Identifiable Reporter”… irrelevant?
•Very little difference between bottom three categories.
The only real change was seen with “strict” definition,
which dramatically reduced hits.
•For Facebook and Twitter, a valid email address is a
requirement, thus rendering a distinction between
“lenient” and “low” meaningless.
•This may point to the discord between a definition
created before the world wide web existed, applied to
social media today.
20
?

21. “Identifiable Event”…subject to change
•Vendors varied in their operational definition of
an “event” based on their specific assumptions
and working models.
•In one instance the vendor used definitions
already developed and used in their daily
business – with the effect that the precision was
high but sensitivity poor.
•What events qualified as hits also varied with
lexicon use and sophistication, and with
application of curation.
21

22. Each Platform has unique considerations
•While it may seem obvious upon reflection, it is
important to realize that searching “social media”
reduces to a collection of specific methodologies for
each platform: Facebook vs Twitter vs Drug Info Sites vs
Patient engagement sites, etc.
•Platforms’ influence was found not only in how data was
collected, but how much could be collected (directly
from API, from 3rd
parties, stored by vendor)
•Twitter provided the most initial data to review, but it
also provided fewer hits per unit number examined
22

23. Curation / Machine approach Tradeoff
• In instances where human curation was used, the precision and
specificity improved (although in two different vendors the
curation method was not comparable)
• However, drugs with a very large initial hit rate (e.g., aspirin,
nicotine) broke the human curation steps, and required various
workaround both in machine and human approaches
• There is as of yet no solely-machine-based approach that
approximates human curation, but there is no human curation
approach that can handle very large numbers in a cost-effective
manner
• This is an area ready for innovation (e.g., a machine-human-based
solution that takes advantage of crowd-sourcing?)
23
vs

24. SUGGESTIONS

25. We need to focus on methods...
25
It is clear that results our
not comparable across
studies or vendors without
significant (re)work
Our enthusiasm to
incorporate SM into PV
has now outstripped our
methods
Value from social media
and PV will come from
generalizable,
reproducible results
It’s time to adopt an
“open-science” style
approach to social media
PV research

26. “Open Science”
“...is the movement to make scientific research, data and
dissemination accessible to all levels of an inquiring society,
amateur or professional. It encompasses practices such as
publishing open research, campaigning for open access,
encouraging scientists to practice open notebook science, and
generally making it easier to publish and communicate
scientific knowledge.”
Wikipedia
26

27. Five suggested areas of methodological
focus
•Operational definitions of drugs, terms
•Lexicon use, sophistication
•Sourcing and amount of data collected
•Algorithms used
•Type and extent of curation
27

28. All PV studies with Social Media Should
Publish the following along with results...
•Operational definitions of drugs, terms
•Lexicon use, sophistication
•Sourcing and amount of data collected
•Algorithms used (to the extent possible)
•Type and extent of curation
28
!

29. And lastly, it would be good to agree...
29
•For reporting on safety of drugs/devices to regulatory authorities
involving social media, more transparency and sharing of methods is
highly desirable
•An industry-agreed “good social-media / PV research practices” that
goes beyond the current general recommendations, and begins to
address those areas raised here, would be timely
•A re-examination of the regulatory definitions of the “4 elements” is
needed, to ground them in modern concepts that include social
media and the internet – a greater specificity in regulatory
definitions will ease the burden on the reporter and lead to greater
standardization in methods

30. Thank You