This document summarizes a data engineering project for analyzing trending topics by geo-location in 3 sentences or less: The project involves building a pipeline to ingest real-time social media data from Kafka into HDFS for batch processing with Spark and storing results in Cassandra, with the goal of exposing trending hashtag data via a web API. Some initial components including a simple Flask API are complete, while work remains on real-time streaming, a NoSQL database interface, and fully configuring the cluster. The presenter has a computer science degree and experience as a software engineer at Citrix and a university research center.

1. Trending Topics
by Geo-Location
Insight Data Engineering
Tigran Antonyan
New York

2. Demo!!!
API/Web Form Works!
• Only the a simple API/Form works so far
‣ via Flask
• http://trendinghashtags.net:5000

3. Data Transformation
Map ReduceFilter Map Map Reduce

4. Data Flow / Pipeline
Message
Broker
Real-Time
Streaming
DB
No-SQL
UI
Web-based
Camus
Batch Processing
Some work
remains
❌
❌
❌

5. Cluster Setup
HDFS Namenode
Camus (cron) /
Kafka Consumer
Spark Master
Kafka Broker
HDFS Datanode #1
Stream Reader /
Kafka Producer
Kafka Broker
Spark
HDFS Datanode #2
Kafka Broker
Spark
HDFS Datanode #2
Kafka Broker
Spark
Cassandra Seed
Cassandra
Cassandra
m4.large nodes m3.medium nodes

6. About Me
Camping
• BS in CS at FAU
• Software Engineer
‣ Citrix
‣ UCONN VoTeR Center