This document discusses trending topics by geo-location using data from Twitter. It describes the data flow and pipeline, including streaming tweets from the Twitter API to Kafka and processing them with Spark on HDFS for hourly and daily trends. The cluster setup is outlined showing the various components. Challenges around scaling to millions of tweets per day are discussed, requiring upgrades to memory and server sizes. Storm topologies are used to consume from Kafka, write tweets to storage, and aggregate minute-based trends for a live page. Time discrepancies between servers are also noted.

1. Trending Topics
by Geo-Location
Insight Data Engineering
Tigran Antonyan
New York

2. Demo!
• TrendingTopics.info
‣ Source: Twitter Streaming API
‣ Topic is a #hashtag or a user mention
- Live page video

3. Data Transformation
Map Reduce Filter MapReduce

4. Data Flow / Pipeline
Message
Broker
Real-Time
Streaming
DB
No-SQL
Web-based UI
Camus
Batch Processing
〜～4M tweets / day
〜～1.7M with tags
Daily
Hourly
Real Time
Tick Tuples (1m)

5. Cluster Setup
HDFS Namenode
Camus (cron) /
Kafka Consumer
Spark Master
Kafka Broker
HDFS Datanode #1
Stream Reader /
Kafka Producer
Kafka Broker
Spark
HDFS Datanode #2
Kafka Broker
Spark
HDFS Datanode #2
Kafka Broker
Spark
m4.large
m4.large
Cassandra
Cassandra Cassandra
UI / Flask
t2.micro

6. Challenges/Issues
Scale
• JVM ran out of memory when Storm
consumed all available tweets from Kafka
• Cassandra crashed a few times
‣ ulimit needed to be increased
‣ JVM heap size needed to be increased
‣ Upgraded from m3.medium to m4.large
• UI server runs out of memory when dealing
with large number of results (100’s of K)

7. About Me
• Software Engineer
‣ Citrix
‣ UCONN VoTeR Center

8. Storm Topology
• Spout:
‣ Kafka consumer
• Real Time Bolt:
‣ Write tweet basic information (time, text, etc.)
‣ Write tweet locations (lat., long.) if available
• Tick Tuple Bolt:
‣ Aggregate minute based trends for live page

9. Time Discrepancies
• 30+ seconds difference between the servers.