The Twitter data firehose delivers hundreds of millions of Tweets every day. This data flood comes with many ‘big data’ challenges in terms of both data volumes and velocities. This presentation will focus on tools that help you find your data ‘signal’ of interest, and will include several demos that focus on using Twitter for flood early-warning systems. These demos will highlight the real-time, public broadcast nature of Twitter, examples of real-time firehose filtering, as well as recent Internet of Things (IoT) Twitter integrations.

1. Floods of Twitter Data
Jim Moffitt
Developer Advocate @Twitter
@snowman
StampedeCon 2016

2. Gnip develops the enterprise-grade data APIs and services to
help unlock the power of Twitter data

3. agenda
(6 characters)
• Twitter 101 - Public, mobile, real-
time and big data.
• Finding your signal of interest.
• Twitter, early-warning systems
and IoT.

4. Twitter is
public, mobile, and
real-time
(40 characters)

5. Firehose volume:
• Hundreds of millions of
Tweets every day.
• Billions of Tweets per
month.
Twitter is big data

6. Twitter is big data
Firehose velocity:
• Many thousands of
Tweets per second.
• 20-25K per second
now common during
events.
• Current record: 140K
tweets per second.

7. Twitter is real-time

8. Twitter is really fast
#StampedeCon OR #realtime OR @snowman OR
(point_radius:[-90.202222 38.624419 1mi] (@StampedeCon OR
#demo OR "big data" OR #BigData))

13. Publisher Data Platform
Sample Analytics Providers
Brands
Acting on Insights from Twitter Data
Providing Analysis
from Twitter Data
Filtering, Enriching
& Distributing
Content
Creating Social
Media Content
TWITTER DATA ECOSYSTEM

14. SYSTEM
REDUNDANCY
CONFIGURATION
APPLICATION
FILTERING AT
SCALE
DATA FIDELITY
Enterprise Data Services

15. FIREHOSE
FILTERING
STREAMING
CLIENT SIDE
RULES

16. FIREHOSE
FILTERING
STACK 1
CUSTOMER
STREAMING
STACK 2 STACK 3
CUSTOMER 1
CUSTOMER 2
CUSTOMER 3
CUSTOMER 4
CUSTOMER 5
CUSTOMER 6
CUSTOMER 7
CUSTOMER 8

17. • Agnostic APIs.
• Most used languages: Java, Python, Ruby, C#, Node.js
• Understanding Twitter metadata and filtering syntax.
• Stream consumer is lightweight and writes Tweets to a queue.
• JSON parser is flexible and tolerant.
• Bending and not breaking.
• Storage.
Client-side details

18. A Twitter Data Story
2013 Colorado Flood

19. Public Broadcasts
Tweeting Alerts and Notifications

20. BoulderCreek
StreamLevel(Feet)
2.25
4.5
6.75
9
Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17
HourlyRainfall
(Inches)
0.3
0.6
0.9
1.2
Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17
TweetsPerHour
1500
3000
4500
6000
Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17
Major Flooding
Moderate Flooding
Stream Bank Full
Event Tweets
Baseline Tweets

21. Audiences tune in during events…

22. TotalRain(Inches)
4
8
12
16
Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17

23. NewFollowers
1500
3000
4500
6000
Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17

26. IoT Examples

36. Jim Moffitt @snowman
Questions?