Python Web-scraping and Data Analysis project as part of online bootcamp conducted by NYC Data Science Academy

1. Naribole
Dec 14, 2016
1
/r/Soccer Activity and User Distribution
Analysis
Project Web Scraping
Sharan Naribole

2. /r/Soccer Introduction
1
• Large community
o Over 500,000 subscribers globally
o Hundreds of posts discussed daily
o Goals, pre, post and live match discussion, news articles etc.

3. /r/Soccer Flairs
2
• User Flair
oTeam crest appears beside username on all comments
o Provides additional dimension of context to the comments

4. Objective
3
To scrape and analyze /r/soccer top posts for:
a) user flair distribution and
b) it’s relationship with comments activity, submission score and type

5. Outline
4
Data Collection
Data Processing
Data Analysis

6. Data Collection
5
• Every post in the top 1000 posts during Nov 12 2016 -Dec 12 2016
o Constraint: Minimum of 100 comments
o Matches occur on a weekly basis!
o Over 500 posts/submissions met the constraints
• Per-submission Information
o Title
o Submission score (~= Upvotes - Downvotes)
o Number of comments
o Unique user-flair mapping in top 500 comments

7. Data Collection Tools
6
• Scrapy Crawl Spider
o /r/soccer home page sorted by top
o Comments page of each submission
• SelectorGadget

8. Per-Submission Computation
7
• Flair Diversity
o Unique number of flairs
• Percentage share per flair
o 100* Number of commenters of a flair/ Total commenter-flair mappings
• Top percentage share
o Highest percentage share among all the flairs
• Comments
o Language processing to find total number of comments

9. Computation Framework
8
Title
Flair
Diversity
Top Share Comments
Submission
1
Submission
2
Submission
3
…
Submission 1 Submission 2 …
Flair 1
Flair 2
Flair 3
…
• Pandas DataFrames

10. Data Analysis
9
Diversity - Comments - Score Relationship
Submission Type Analysis
Flair Distribution

11. Diversity - Comments - Score Relationship
10
• Comments increase with Diversity
• Submission Score increases with Diversity

12. Submission Type Analysis
• Goal video submission
o Posted in near real-time with high quality goals getting top score
o Hypothesis:
★ Flair diversity as goals are discussed for their quality
★ Comments proportional to score
• Match Thread, Post-Match Thread submissions
o Discussion during and after live match
o Hypothesis:
★ Low flair diversity as teams taking part in the match expected to have high share
★ High number of comments as users comment on various events not just goals
11

13. Submission Type - Comments Analysis
• Expectedly, comments are higher for Match Threads
12

14. Submission Type - Flair Diversity Analysis
• Unexpectedly, Match Threads have higher flair diversity
• Match Threads among top posts likelier discussed by other fans
13

15. Submission Type - Score Analysis
• Match and Post-Match Threads most active during and just after a match
• Goal submissions are rated for the quality of goal increasing over time
14

16. Flair Distribution
• English Premier League clubs lead the /r/soccer table!
15

17. Conclusion
16
• Scraped, visualized and analyzed /r/soccer top posts during past one month
★ Flair Diversity, Score and Top Flair Share relationships
★ Submission type-analysis
★ Flair distributions