The first indication of a new outbreak is often in unstructured data (natural language) and reported openly in traditional or social media as a new ‘flu-like’ or ‘malaria-like’ illness weeks or months before the new pathogen is eventually isolated. We present a system for tracking these early signals globally, using natural language processing and crowdsourcing. By comparison, search-log-based approaches, while innovative and inexpensive, are often a trailing signal that follow open reports in plain language. Concentrating on discovering outbreak-related reports in big open data, we show how crowdsourced workers can create near-real-time training data for adaptive active-learning models, addressing the lack of broad coverage training data for tracking epidemics. This is well-suited to an outbreak information- flow context, where sudden bursts of information about new diseases/locations need to be manually processed quickly at short notice.

1. ‘Tracking Epidemics with Natural
Language Processing and
Crowdsourcing’
Robert Munro, Lucky Gunasekara, Stephanie
Nevins, Lalith Polepeddi and Evan Rosen
Stanford University and EpidemicIQ
2012 AAAI Spring Symposium
March 2012
http://www.robertmunro.com/research/mu
nro12epidemics.pdf

2. Global Viral Forecasting
weakly human
adapted
human adapted
human exclusive
Influenza HIV-1Yellow FeverRabies SARS/Ebola
transmissible
not human
adapted

3. 90% of the
world’s
ecological
diversity
90% of the
world’s
linguistic
diversity

4. Reported
locally before
identification
H1N1 (Swine Flu) –
months
(10% of world infected)
HIV – decades
(35 million infected)
H1N5 (Bird Flu) –
weeks
(>50% fatal)

5. Diseases eradicated in the
last 75 years:
Increase in air travel in the last
75 years:
s m a l l p o x

6. No one is tracking all the world’s outbreaks
• NASA is tracking thousands of potentially
dangerous near-Earth objects (NASA 2011).
• National security agencies are tracking tens
of thousands of suspected terrorists daily
(Chertoff 2008).
• A deadly microbe is far more likely to sneak
onto a plane undetected.

7. CDC vs Google Flu
Trends?

8. CDC vs Google Flu
Trends?
Source: http://www.google.org/flutrends/

9. CDC vs
Google Flu
Trends?
"I'm Jacqui Jeras with
today's cold and flu
report ... across the
mid- Atlantic states, a
little bit of an increase
here” Jan 4th

10. "I'm Jacqui Jeras with
today's cold and flu
report ... across the
mid- Atlantic states, a
little bit of an increase
here” Jan 4th
CDC vs
Google Flu
Trends?

11. The first signal is plain language
“today's cold and flu report ... across
the mid-Atlantic states, a little bit of an
increase” CNN
Jan 4, 2008
Google Flu Trends
+ 3 weeks
CDC
+ 5 weeks

12. … but buried in plain view
“today's cold and flu report
... across the mid-Atlantic
states, a little bit of an
increase”
“We're worried about the
markets.”
“We're going to take you to
Kenya where the U.S. has
dispatched some diplomatic
help to try to get the
country back on political
balance.”
“Is individualism an
endangered concept
in Saudi Arabia?”
“Well, in St. John's
County, one man lost
his home trying to
keep his pig warm.”
“The pig did not
make it.”
“He had everything but
the cape. A good
samaritan in Ohio saved
a family from this
ferocious house fire.”
“A spunky boy reels
in a 550-pound
shark.”

13. … in 1000s of languages
в предстоящий осенне-зимний период в
Украине ожидаются две эпидемии гриппа
(2 flu outbreaks predicted for the Ukraine)
‫ر‬‫ص‬‫م‬ ‫ي‬‫ف‬ ‫ر‬‫و‬‫ي‬‫ط‬‫ل‬‫ا‬ ‫ا‬‫ز‬‫ن‬‫و‬‫ل‬‫ف‬‫ن‬‫ا‬ ‫ن‬‫م‬ ‫د‬‫ي‬‫ز‬‫م‬
(more flu in Egypt)
香港现1例H5N1禽流感病例曾游上海南京等地
(Hong Kong had a case of avian influenza that
traveled to Shanghai and Nanjing)

14. M a c h i n e -
l e a r n i n g :
R e l e v a n t ?
R e p o r t s
( m i l l i o n s )
в предстоящий осенне-
зимний период в Украине
ожидаются две эпидемии
гриппа
‫ن‬‫م‬ ‫د‬‫ي‬‫ز‬‫م‬‫ز‬‫ن‬‫و‬‫ل‬‫ف‬‫ن‬‫ا‬‫ي‬‫ف‬ ‫ر‬‫و‬‫ي‬‫ط‬‫ل‬‫ا‬ ‫ا‬‫ر‬‫ص‬‫م‬
香港现1例H5N1禽流感病例曾
游上海南京等地

15. Targeted machine-
processing
Broad machine-
processing
Human-
processing
Low-volume
processing
High-volume
processing
Data input
“there is a new flu-like illness here”
Discovered by
crawler
Relevance
evaluated by
machine learning
Relevance
evaluated by
microtasker
Information stored
from the reports
Relevance
evaluated by in-
house analyst
Sources monitor-
frequency
updated
Maximally relevant
phrases used to
search more data
Direct report from
field staff / partner
organization
Reports for each
outbreak
aggregated

16. Data structuring
• Disease (if known)
• Case counts / demographics
• Location
• Responding organizations
• Transport used
• Quotes from officials
• Changing conditions (spreading / ending)
• Public reaction

17. Motivations
For 600 new seeds, please answer this question:
Does this sentence refer to a disease outbreak:
“E Coli spreads to Spain, sprouts suspected”
Yes/no: __
What disease: _______
What location: _______
Virtual protecting the real

18. E Coli, Germany 2011
The AI head-start

19. Predicting epidemics, 100K training items

20. Crowdsourcing applicability
• Success:
– Language coverage
– Outbreak relatedness
– Case-counts
– Location names
– Quotes from officials
• Falling short:
– Estimating citizen unrest
– Growth predictions
Native speaker
expertise
Data structuring
Data analysis

21. Crowdsourcing and machine-learning
• The German problem
• Bias-free seeding
• Evaluation for needle-in-haystack scenarios
– Machine and human
• Language representation

22. Acknowledgements

23. Questions?

24. Appendix: Abstract
The first indication of a new outbreak is often in unstructured data
(natural language) and reported openly in traditional or social media
as a new ‘flu-like’ or ‘malaria-like’ illness weeks or months before the
new pathogen is eventually isolated. We present a system for
tracking these early signals globally, using natural language
processing and crowdsourcing. By comparison, search-log-based
approaches, while innovative and inexpensive, are often a trailing
signal that follow open reports in plain language. Concentrating on
discovering outbreak-related reports in big open data, we show how
crowdsourced workers can create near-real-time training data for
adaptive active-learning models, addressing the lack of broad
coverage training data for tracking epidemics. This is well-suited to
an outbreak information- flow context, where sudden bursts of
information about new diseases/locations need to be manually
processed quickly at short notice.