Summary of my demo project for the Insight Data Science Fellows program. **Find relevant content to plan your next hobby or home improvement**

1. prep fo/r/diy
Find relevant content to plan your
next hobby or home-improvement
Kristofor Nyquist

2. …but reddit is a popularity contest
Given that I’m interested in a post
1. What are similar posts I can read?
2. Who are the best people I can talk to?
Goal: find related DIY projects

3. Collecting / organizing data
• Scraped reddit for do-it-yourself projects
– Collected text content from each project
i.e. title, externally linked blog post, comments,
and general topic

4. Data conversion
• Combine all of the text to create one
“document”
• Treat the document as a list of words
• Convert the list of words to a list of numbers
– Each number represents the “uniqueness” of a
particular word to its document
i.e. 0 means word appears in every post
1 means word only appears in that post

5. Data conversion
• Combine all of the text to create one
“document”
• Treat the document as a list of words
• Convert the list of words to a list of numbers
– Each number represents the “uniqueness” of a
particular word to its document
i.e. 0 means word appears in every post
1 means word only appears in that post

6. Data conversion
• Combine all of the text to create one
“document”
• Treat the document as a list of words
• Convert the list of words to a list of numbers
– Each number represents the “uniqueness” of a
particular word to its document
i.e. 0 means word appears in every post
1 means word only appears in that post

7. Data conversion
• Combine all of the text to create one
“document”
• Treat the document as a list of words
• Convert the list of words to a list of numbers
– Each number represents the “uniqueness” of a
particular word to its document
i.e. 0 means word appears in every post
1 means word only appears in that post

8. Data conversion
• Combine all of the text to create one
“document”
• Treat the document as a list of words
• Convert the list of words to a list of numbers
– Each number represents the “uniqueness” of a
particular word to its document
i.e. 0 means word appears in every post
1 means word only appears in that post

9. Clustering
Text content is rich enough to cluster projects

10. Clustering
Classify projects with missing categories based on user inputs

11. Validating post similarity
Similar posts group well by general topic.
Deviations can make sense

12. Against random
Obvious that randomly picking posts is not suitable

13. PhD Biophysics, UC Berkeley
BS Physics, WSU
About me (Kristofor Nyquist)
Hobbies

14. Algorithm
Post similarity / Classification
• Turn the text into a list of numbers using term-frequency-
inverse-document-frequency
• “Compress” the data for speed
– ~70,000 dimensions to 80 dimensions
For similarity:
• Calculate cosine-similarity between documents
• Present user with 5 most similar posts
For classification:
• Logistic regression (L1 regularization)

15. Settling on 80 PCs
80 principal components somewhat arbitrary
BUT overall accuracy of classifier has definitely converged…
even though 80 PCs capture ~30% of variance

16. Validation numbers
• NLP algorithm
– accuracy: 0.62
recalls:
auto: 0.73
electronic: 0.6
home improve: 0.54
metalwork: 0.44
other: 0.52
outdoor: 0.89
woodworking: 0.68
• Random
–accuracy: 0.17
recalls:
auto: 0.05
electronic: 0.00
home improve: 0.18
metalwork: 0.00
other: 0.08
outdoor: 0.12
woodworking: 0.32

17. Full BoW vs. 80 PCs
80 PCs Full tfidf vector

18. Validating classifier
auto
metalwork
home
electronic
other
outdoor
woodwork
auto
metalwork
home
electronic
other
outdoor
woodwork
Prediction
Truth
Column normalized
vs. rest ROCWoodworking
Other
Home improvement
With this application, can tolerate false
positives. We can also present users
with more limited options,
maintaining higher accuracy