Yun Yuan is developing a web app to analyze how opinions evolve over time in Reddit comments. The app will organize comments for each post in timestamp order and identify hot comments. Reddit comment and post data will be ingested from JSON files into Hadoop and analyzed using Spark to identify trends. The results will then be presented on a Flask web app. Challenges include handling null values in the JSON data and varying comment trends. The project uses distributed computing clusters with Hadoop, Kafka, Cassandra and Spark to process the large Reddit data at a cost of around $400 per month.

1. Reddit Story
What will happen as the comments
of the post keeps going
Yun Yuan

2. Motivation for Project
• Interested in Social News: All topics under the
sun
• From tree structure of comments to timeline
structure of comments
• See how opinions evolve as time flows

3. Input and Output
Data Input
• Reddit Comments from S3 Data Dump (JSON files)
• Reddit Posts Info from Reddit API (JSON files)
Data Output
• For each post, organize comments in timestamp- base
with some significant attributes, and show hottest
comments for that post
• Web App Presentation (Link): Graph and Short Texts
• Demo: Under Construction

4. Tentative Pipeline and Data Flows






+

Comments
JSON
Posts
JSON
Post -> Trends
-> Hot Comments Challenges Encountered:
• Null Value of field
from JSON when
doing ingestion
• Comment trends vary

5. Distributed Clusters



1 Cluster: 4 Nodes of m4.large
Hadoop/HDFS
Kafka/Zookeeper
Cassandra
1 Node of t2.micro
Flask
1 Cluster: 4 Nodes of m4.large
Spark
~$400 per mon

6. About me: Yun Yuan