PoC to tap into Twitter data and report on diseases. Amed to give indicative, early warning information about communal diseases

1. Presented by: Thomas Otto
Manager Business Intelligence @ ESR
Institute of Environmental Science & Research Ltd.
Credits:
ESR - Franco Andrews: SAP Data Integration, modelling and visualisation
ESR IT and Health Intelligence: Health IP, Infrastructure / Firewall
Soltius NZ - Erik Roelofs: Connection Module and SAP Data Services
Twitter Social Media Analytics
Proof of Concept
© ESR 2015

2. What is Social Media?
Social media refers to the means of interactions among people in
which they create, share, and/or exchange information and ideas in
virtual communities and networks¹.
1 Tufts university, Boston, U.S.A.
2 Social Media Examiner: 2014 Social Media Marketing Industry Report

3. What is Twitter?
1 Wikipedia
2 PEW Research Centre: January 2014
3 DMR: March 2014
4 Twitter Terms of Service as of 24/7/14
Some Twitter statistics
• 1 billion users registered
• 255 million users/month
• 100 million users per day ³

4. Commercial / Government
clients• This is a proof of concept (POC)
• The POC is not used to monitor actual diseases
• This POC is an experimental application at ESR to understand the
validity of the approach
Social Media (Twitter) Analytics overview

5. Commercial / Government
clients
Social Media (Twitter) Analytics overview

6. Social Media (Twitter) Analytics vs. BI / DW

7. Social Media (Twitter) – Visualisation / Front-End
All
keywords
Interactive
map
(Bing)
Basic stats
by location
Tweets
Current,
active
keywords

8. Social Media (Twitter) – Visualisation / Front-End
Select
keyword
Measles
Zoom into
WLG
Basic stats
by location
Measles
Tweets
Current,
active
keywords

9. Social Media (Twitter) – Visualisation / Front-End
Select
Measles
to drive the
dashboard
Measles
Tweets in
Jun : 125
Jul : 196
Measles
Tweets in
WLG : 73

10. Social Media (Twitter) – Visualisation / Front-End
4th of July
Apple
devices are
used most
5 am busy
tweeting…
936 of 10,412
Tweets in
July of
71,719 total
collected
Tweets so far

11. Social Media (Twitter) – Visualisation / Front-End
Measles in
NZ
Strong,
negative
sentiment in
Jan, Jun/Jul

12. Challenges and learnings
- Protect against threads from the cloud
vs. open to the cloud (Twitter)
- Re-configure ESR firewall to allow
connection to Twitter API¹
- Patch BI application with new Python
version to accommodate ESR’s
firewall set-up
- Lots of back & forth, complex, but
eventually fixed issues
1 Application Programming Interface

13. Challenges and learnings
- Learn Twitter API, its settings,
parameters and its limits
- Free TwitterSearch API is time-boxed
and limited in volume
- API 1.1, see https://dev.twitter.com
- Alternatively use interface which
exactly mimics API but with user
interaction, see
https://twitter.com/search-advanced

14. Challenges and learnings
- Learn Python (programming) and
integrate into BI/DW application servers
- Execute Python from within the
ETL-engine¹
- Handle the messaging between
ETL-engine and Twitter API
- Error handling, administer
tokens, messages, counters,
looping etc.
1 Extract, Transform and Loading

15. Challenges and learnings
- Use BI / DW technology to analyse
unstructured (text) data
- Categorise tweets based on a
dictionary and use the Entity
Extraction Transform (EET) of
Data Services (i.e. group
variations of spelling of a
keyword)
- Perform sentiment analysis on
Tweets using EET module
(I am awfully sick = strong, I
am hardly ever sick = weak)

16. Challenges and learnings
- Test new visualisation technology:
Microsoft PowerPivot / PowerView in
Office2013
- Great opportunity to test a different
Business Intelligence / visualisation
tool (ESR uses Business Objects from
SAP for reporting)
- Integration of Bing/maps; no need of
a GIS system at ESR. Out-of-the-box
(provided connected to the internet
to reach Bing server).

17. Probabilityofatweetbeingcorrelatedtoa“realfact”
100 %
0 %
Time
? %
Unverified
Twitter
data
Competitor
ESR
Inject
Health Line
data into ESR
Twitter data
Inject volume related data (count of disease in point of time and location).
Inject
Health Stats
data into ESR
Twitter data
Inject
LIMS
data into ESR
Twitter data
Inject
Sentinel
data into ESR
Twitter data
Inject
Epidemiology
data into ESR
Twitter data
Competitor
ESR
A research letter on the
internet flu tracker in the
scientific journal Nature,
written by officials from
Google and the US
Centres for Disease
Control and Prevention,
says there is a close
correlation between the
rates of doctor visits for
flu symptoms, and the
use of flu-like search
terms. NZ Herald 23/7/14
Google Flu Tracker has overestimated
for 100 of the 108 weeks starting
from August, 2011, according to
the Science analysis, by researchers
at Northeastern University and
Harvard University. source: motherboard.vice.com
VISION - Big Data complements traditional methods
Calibrate social media data with verified and trusted data to identify valid tweets
Risk Opportunity
Increase probability of
success of a campaign
based on Social Media data.
Increase confidence to
invest money in to the right
actions.
Increase confidence in
Social Media data as it’s
verified against other,
trusted sources.
Google ??
Flu Tracker
NZ

18. Would I do the Social Media project again?
• Yes!
• ESR has data to complement Social Media and increase data
quality (verify Tweets).
• Try below tools first and see what benefit they offer…
• PlusOne Social
http://plusonesocial.com
• Microsoft Excel Twitter add-in:
http://www.microsoft.com/en-
us/download/details.aspx?id=26213
Others
• http://www.razorsocial.com/free-twitter-analytics/
• http://www.socialmediaexaminer.com/6-twitter-
analytics-tools/