This talk was presented at the Advances in Precision and Personalized Medicine Workshop.

1. Crowdsourcing for patient and
physician medical insights
Jeffery L. Painter
jeff@jivecast.com
March 14, 2019

2. My Background
• Jeffery L. Painter, BS, JD
– CEO and Founder of Jivecast, specializing in developing
strategies, software and analytics tools for a broad range
of industries since 2016
– Background in NLP, software development, data mining
and biomedical ontologies (UMLS)
– Formerly with GSK (over 10 years), last holding the
position of Director of Analytics in US Medical Affairs
– Recently joined a new, biotech startup: Flagship Labs 60,
Inc in Cambridge, MA

3. How did we get here?
• -2004: state of the art for signal detection was
monitoring spontaneous event data (self
reporting systems, no way to know how many
people are taking a drug or product)
• -2006-2007: worked to develop a common data
model for observational data (EMR/EHR) evolved
into OMOP and adopted by Sentinal
• 2009-2010: can we add social media to decrease
lag time to signal identification à too much data!

4. Objectives
• Crowdsourcing Overview
• Examples of use
• Review key limitations/risks
• Discuss potential uses and costs

5. Crowdsourcing Overview

6. • Crowdsourcing (“think of Uber for intellectual activities”)
– The practice of obtaining needed services, ideas, or content by soliciting contributions
from a large group of people and especially from the online community rather than from
traditional employees or suppliers.
• 3 types of “Crowdsourcing” crowds
1. General Population (Amazon Turk)
2. Disease Specific (e.g. Inspire, PatientsLikeMe)
3. Physician Specific (SERMO)
Overview of Crowdsourcing

7. • Amazon Turk is the largest vendor (users are called Turkers) https://www.mturk.com/
– Amazon mTurk allows for quick, cost effective delivery of HITs (Human Information Tasks)
to Turkers (“workers”)
– Turkers self-select and opt-in to participate in the study
• 500,000+ anonymized Turkers worldwide
– ~50% from the US, use outside of US is quickly expanding
– 5,000 – 7,000 workers online at any given moment
• Identity verified by Amazon (e.g., social security # in US)
• Ability to target individuals on a variety of demographics
– Age, gender, ethnicity, education level, geography
• Substandard work can be rejected
Overview of Crowdsourcing

8. Patient Specific Crowdsourcing
• 3 largest sources (Inspire, HealthUnlocked, Patientslikeme)
• Inspire (https://www.inspire.com/)
• 1,200,000 members
• 221 specific disease communities
• 10,000,000 annual visitors
• Members pre-consented to being contacted by Pharma for R&D
8

9. • SERMO (http://www.sermo.com/)
– Physicians only - credentials are triple verified before allowed to join
– Members from 150 different countries
– 800,000 members
– Identity is anonymized, but SERMO offers us the ability to target by specialty and country
– 75% male, 25% female
– 40% primary care, 60% specialists
– Average 15-20 years in practice
– 60% log in via mobile
– 96 specialities and sub-specialties
– They handle payment reporting (e.g., Sunshine Act)
Overview of Crowdsourcing

10. Crowdsourcing Examples

11. Crowdsourcing Examples
• Noise removal for social listening activities
• Assist in natural language processing capabilities
development
• Manual classification of social media data (from Twitter,
Reddit, etc.) for multiple projects
– Pregnancy health outcomes research
– COPD drug treatment/comparison study
– Identification of potential drug abuse (collaboration with US-
ADA/W-ADA antidoping associations)
• Surveys
– Minority attitudes towards CT participation (US/Global audience)
– Assess cold sore treatment options (UK audience)
– Pirinase Study to help meet regulatory filings (UK audience)
– Plain Language Summary reviews (Adults over age of 18)

12. Examples: Classification
– Problem: Identify potential AE mentions in social media
– Goal: Build a machine learning classifier that could screen social media
for potential AE in real time
– Medical experts manually review around ~10,000 posts identifying 22
medical concepts, then compared those results to Amazon turk workers
Posts (n) Total Questions Matched
Percent
Match
Curation
Time(Hours)
Phase 1
$0.03 per Post 500 11,000 10,217 92.8% 147
$0.03 w/bonus 50 Posts 500 11,000 10,216 92.8% 147
$0.05 per Post 500 11,000 10,189 92.6% 146
Phase 2
$0.18 per Post 5,000 110,000 100,981 91.8% 33

13. Prior Study Examples: PLS Review
• Plain Language Summary (PLS) Feedback
– Sought PLS feedback from multiple targeted groups
– Survey programming took a couple of hours
– 100 surveys completed (5 PLS, 20 surveys each)
– Average time to complete survey was 1 hour
– Total project completed in a single evening
– Cost was $10.00 per survey ($1,000 USD total)
• Stakeholder Feedback
– “This system offers a fast way to see if specific portions of a PLS
are understandable”
– “Great method for quick and inexpensive check where needed”

14. Prior Study Examples: PASS
• Team wanted to evaluate if crowdsourcing could replace a failing pass
(Post-authorization safety study)
• Current study by CRO had enrolled 49 patients in 2 years at a cost of
~£150,000.
• Exact same study survey posted on Amazon Turk (UK Turkers only)
• ~120 people completed survey in first 3 months , 300 completed by
December 2017
• Paid participants $1.00 per survey ($350 USD total)
• STP submitted to MHRA to get approval for crowdsourcing to replace
CRO for PASS
• Stakeholder Feedback
– “I have been so impressed with the output, I have recommended
my other project teams to come to you for support”

15. Examples: Pregnancy and Lupus
• Can we identify women who have recently been or are
currently or planning to become pregnant with diagnosis of
Lupus?
• Highly targeted population
• Pregnancy status can change over time

16. Methodology
• Our patient survey stated the
following requirements:
1. Female between 18-55 years
2. With a diagnosis of SLE
3. And have been pregnant (past two
years) or planning or trying to
become pregnant
– Survey ran approximately 3
months (Jun 11 – Sep 17,
2018) with steady
submission rate
– 151 total responses
collected, primarily from US
– Average time to complete each survey was 9
minutes and participants received $1.00 (USD)
compensation for this survey
16
Crowdsourcing for the BPR awareness study
Rate of Patient Survey Submissions
0
20
40
60
80
100
120
140
160
6/11/2018
7/11/2018
8/11/2018
9/11/2018
TotalSubmissions
Survey Completion Date
Total Submissions

17. Testing veracity of the Data: Treatment Patterns
17
How do we know the data is accurate?
Comparing treatment options as recommended by HCPs and what patients report
Survey Results correspond to clinical knowledge
1. Belimumab is recommended as a treatment more commonly for patients who exhibit severe symptoms rather than mild
2. Patients reported use of belimumab by disease severity tracks in near, lock step with HCP recommendations
0%
20%
40%
60%
80%
100%
120%
NSAIDCorticosteroidsAntim
alarlials
Im
m
unosuppresiveagents
belim
um
ab
rituxim
ab
Mild
Moderate
Severe
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
NSAIDCorticosteroidsAntim
alarlials
Im
m
unosuppresiveagents
belim
um
ab
rituxim
ab
Mild
Moderate
Severe
HCP Recommended Treatment Options Patient Reported Treatment Options

18. Key Limitations and Risks

19. Crowdsourcing Limitations/Risks
• Getting the questions right – are we are getting the
information needed and not introducing bias?
• Veracity of the data – are the people responding really your
target population?
• Acceptance by regulatory agencies –how will regulatory
agencies view/accept this emerging capability?
• Proper utilization – will teams properly
contextualize/utilize/interpret crowdsourcing results?

20. Legal Considerations
• We recommend that each team seek whatever legal
approvals they feel may be needed prior to engaging
directly with patients via Amazon Turk or similar
platforms

21. Potential Uses and Costs

22. Potential Uses of Crowdsourcing
• Patient insights (disease progression, unmet need, etc.)
• Patient preference (e.g,, self-administered shots)
• Protocol feedback/patient recruiting
• Understanding comprehension of product labeling
• Validation of patient reported outcomes within specific sub-
types (ie, severe refractory asthma)
• Understanding barriers to adherence
• Understanding cultural differences which impact study
participation, and health care beliefs
• Understanding patient and support group interactions related
to providing health care

23. General crowdsourcing: Project Costs
• Dependant on type of audience sought and number
of responses required
• General crowdsourcing: On average, we attempt to
compensate Amazon turk workers at least minimum
wage
– PASS study - $2.00 USD / response
– Cold sore survey – $1.00 USD / response
– PLS Document review - $10.00 USD / response
• We always restrict participants to one response per
survey to prevent double counting

24. Summary
• Crowdsourcing is an emerging technology that offers a lot of
promise.
• We can target the general population, patients with specific
diseases, or physicians with a range of activities (disease
insights, comprehension of study results, etc.)
• Key advantages include: cost, speed, and geographic coverage.
• Risks that must be mitigated include: people properly using and
interpreting the results. Crowd sourcing can supplement your
research, but should not be the only source of data

25. Questions