Nerddit Demo presentation

1. More than a reddit N-gram viewer
JERRY PRAWIHARJO
INSIGHT DATA ENGINEERING FELLOW
nerddit

2. Motivation
N-grams:
Allows Data Scientists to do Topic trends analysis, language analysis
Allows for “Type-ahead” feature
Subreddits network graph:
SR1
SR2
SR3
U
U
U
U

3. 1-gram
My Name is Jerry
“My” “Name” “is” “Jerry”

4. 2-grams
My name is Jerry
“My Name”
“name is”
“is Jerry”

5. 3-grams
My name is Jerry
“My name is”
“name is Jerry”

6. Pipeline
6x m4.xlarge
$1.43/hour
5x m4.xlarge
$28.7/day
t2.micro
free
~10GB
10/2007-12/2015
>1TB uncompressed
4x m4.large
$11.5/day

7. Reddit Statistics
Year Date comments Unique authors Unique subreddits
2015 2015-12-01 10000 25000 60000
2014 2014-12-01 50000 35000 40000

8. N-gram
Ngram Date N Count Percentage
Hallows 2011-04 1 10 0.1
Deathly Hallows 2011-04 2 50 0.1
Ngram N Subreddit Count (counter type)
Hallows 1 movies 1000
Deathly Hallows 2 movies 5000
N gram cluster against subredditsTime series ngrams
(“2011-04”, [“old”, “lady”, .., “Deathly”, “Hallows”,…], “movies”])
(“2011-04::old::movies”, 1)
(“2011-04::lady::movies”, 1)
…
(“2011-04::Deathly::movies”, 1)
(“2011-04::old::movies”, 10)
(“2011-04::lady::movies”, 5)
…
(“2011-04::Deathly::movies”, 2)Job took ~2days to complete
Regex filters
URLs, IMG links,
unicodes

9. Subreddits Graph
Year node1 node2
2011 movies {politics: 10, games:5,…}
2014 politics {games: 3,conservative: 2,…}
Year Distinct authors subreddit Comments
2011 TheOceldoc movies 100
2011 JohnDoe politics 200
(TheOceldoc, (movies, politics, games,…)
(JohnDoe, (politics, conservative,…)
(“movies::politics”,10)
(“movies::games”,5)
(“politics::games”,3)
…
(“politics::conservative”,2)
Edge weight
Filter degree < 100
Clustering
Force Atlas 2 layout

10. Spark Tuning
0
2
4
6
8
10
12
14
16
18
A B C D
Time(minutes)
Case
Case Rdd Compress Kryo
A FALSE FALSE
B TRUE FALSE
C TRUE TRUE
D FALSE TRUE

11. Jerry Prawiharjo
Phd in Optoelectronics from Southampton England
◦ Distributed computation on Beowulf cluster (MPI)
Product Development Engineer at Neophotonics
◦ Test software development and data analysis
Senior Test Development Engineer at Cisco
◦ Test station development (hardware and software)
for 100G transceiver module

12. Back Up

13. Challenges
Sheer amount of Data: >1TB
◦ Scoping the project: monthly time bucket (as opposed to daily or weekly)
◦ Filter foreign language subreddits
◦ Spark tuning
S3 rate limit: Process data on file-per-file basis