Vasudha Reddy of the NYC Department of Health and Mental Hygiene will talk about how to analyze Yelp reviews of New York City restaurants to identify sources of foodborne illness. Watch the video online: https://www.youtube.com/watch?v=AMQWqU8aZWk&list=PL65XgbSILalWFStqV0z0N9pvftstJ8AAh&index=10 Get involved with Code for America: www.codeforamerica.org/action

1. USING SOCIAL MEDIA DATA TO
TO IDENTIFY UNREPORTED COMPLAINTS
AND OUTBREAKS OF FOODBORNE
ILLNESS IN NYC
FOURTH ANNUAL CODE FOR AMERICA SUMMIT
SEPTEMBER 23-25, 2014
SAN FRANCISCO, CA
Vasudha Reddy, MPH
New York City Department of Health and Mental Hygiene

2. FOODBORNE DISEASE
 1 in 6 Americans (48 million) get sick from a
foodborne disease each year nationally
 45% of confirmed outbreaks are restaurant related
 New York City
 24k restaurants and 15k food retailers
 2 of 3 New Yorkers eat out at least once per week
 3000 complaints received via NYC complaint line
(311, NYC non-emergency number and website)
 30 restaurant related outbreaks identified annually

3. UNREPORTED FOODBORNE ILLNESS
 While investigating an outbreak, noted illness
complaints were posted on Yelp, but not to 311
 Online review sites becoming more common
 Collaboration with Columbia University and Yelp to
use restaurant reviews to identify unreported
foodborne illness complaints

4. PROJECT METHODS:
AUTOMATED ANALYSIS
 Extract keywords, detect temporal statements and
multiple illness reports
 Score review on three criteria learned from a training set
of manually labeled examples
 Key words: sick, vomiting, diarrhea, etc.
 Multiple people ill
 Relevant incubation (>10 hours)
 Averaged scores
 Score >0.5 were flagged for human review

5. PROJECT METHODS:
CRITERIA FOR INVESTIGATING FURTHER
 Episode of illness related to a restaurant?
 Did illness occur recently?
 2 or more people ill or
 Severe illness occurred

6. PROJECT METHODS:
DOHMH INVESTIGATION OF REVIEWS
 Contact reviewers to schedule a phone
interview
 Obtain information needed to conduct a
foodborne disease outbreak investigation
 Interview patrons about illness and foods eaten
 Inspect the restaurant and review food handling
practices
 Collect and test patient and food samples

7. SAMPLE REVIEWS
 Review that does not indicate any illness at all
 Review consistent with a foodborne illness

8. PROJECT RESULTS:
JULY 1, 2012 – JUNE 30, 2014
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
N=4227 Possible Foodborne Complaint Reviews
Indicated Food
Poisoning
Required
investigation
6 outbreaks
identified
Investigation
completed
1912
427
54

9. CONCLUSION
 <10% of complaints identified in Yelp were reported
to DOHMH
 Many reviewers unaware of 311 complaint line
 Yelp’s reviews provide information on restaurants
that may pose a public health risk
 Unique because it provides useful information that
allows us to validate the reviews that are pulled by a
text classifer

10. NEXT STEPS
 Add additional sources of social media data
(i.e. Twitter)
 Refine machine learning and analysis
 Continue partnerships with academic, private
and other government agencies

11. ACKNOWLEDGEMENTS
NYC DOHMH
Sharon Balter
Heather Hanson
Cassandra Harrison
Kenya Murray
Faina Stavinsky
HaeNa Waechter
Yelp
Luther Lowe
Columbia University
Luis Gravano
Noemie Elhadad
Daniel Hsu
Mohip Jorder
Henri Stern
Fotis Psallidas
Alden Quimby

12. THANK YOU!
Vasudha Reddy
New York City Department of Health and Mental Hygiene
vreddy@health.nyc.gov
www.nyc.gov/health