● Presentation of Insight Data Engineering Fall 2016 Project ● https://github.com/rheeluke/Hashtag-Hashtag

1. ##
Luke Young Rhee

2. #The trending topics
of trending topics
#
Luke Young Rhee

3. Impact
Discovery

4. #The trending topics
of trending topics
#
htht.tech
DEMO

5. YouTube

6. The Pipeline

7. The Pipeline
{"text": "RT @Pozzzzzzz
…
"id": 5469180,
“time”: “2016:09:30T...”,
"entities": {"user_mentions":
...
"hashtags": [
{“text”: ”yum”,“indices”: [32,
35] },
{“text”: ”beer”, “indices”: [32,
36] }
Tweet

8. The Pipeline kafka-connect-twitter
● Kafka Connect
● Data

9. The Pipeline Kafka Streams
● Process
● Filter

10. The Pipeline
Druid
● Low latency
○ Ingestion
○ Analytics
● Scalable

11. The Pipeline
Query: Table
time hashtags
... [“beer”, “yum”]
... [“beer”, “Lagunitas”]
... [“cats”, “cute”, “notCrazy”]

12. The Pipeline
Query: Filter
time hashtags
... [“beer”, “yum”]
... [“beer”, “Lagunitas”]
... [“cats”, “cute”, “notCrazy”]
... “beer”
... “yum”
... “beer”
... “Lagunitas”
hashtags = “beer”

13. The Pipeline
Query: Count
time hashtags count
... “beer” 2
... “yum” 1
... “Lagunitas” 1

14. The Pipeline
Query: Count
+
TopN
time hashtags count
... “beer” 2
... “yum” 1
... “Lagunitas” 1

15. The Pipeline

16. The Pipeline
Kafka
Connector
Kafka
Streams
Kafka
Indexing
Service
pydruid

17. Challenges
Kafka Streams /
Serdes
+
Druid

18. Druid Cluster

19. Challenges
Kafka Streams /
Serdes
+
Druid
Ingest: 1% Twitter
~ 1k - 2k /min
Query: ~ 92.6k rows
Success!

20. Thanks!
Luke Young Rhee
University of California, Irvine
MS Mathematics
Nintex, Irvine
Test Analyst
Enjoy being near the ocean and
getting lost in new cities