The document outlines Sarah Guido's presentation on utilizing Apache Spark for data science at Bitly, sharing her workflow and the tools within Spark. It emphasizes the need for efficient big data analysis and explores challenges faced in data processing, including exploratory data analysis and topic modeling. The talk also touches on future applications of Spark in production and ongoing projects at Bitly.

1. Data Science at Scale:
Using Apache Spark for Data Science
at Bitly
Sarah Guido
Data Day Seattle 2015

2. Overview
• About me/Bitly
• Spark overview
• Using Spark for data science
• When it works, it’s great! When it works…

3. About me
• Data scientist at Bitly
• NYC Python/PyGotham co-organizer
• O’Reilly Media author
• @sarah_guido

4. About this talk
• This talk is:
– Description of my workflow
– Exploration of within-Spark tools
• This talk is not:
– In-depth exploration of algorithms
– Building new tools on top of Spark
– Any sort of ground truth for how you should be
using Spark

5. A bit of background
• Need for big data analysis tools
• MapReduce for exploratory data analysis == 
• Iterate/prototype quickly
• Overall goal: understand how people use not
only our app, but the Internet!

6. Bitly data!
• Legit big data
• 1 hour of decodes is 10 GB
• 1 day is 240 GB
• 1 month is ~7 TB

7. Why Spark?
• Fast. Really fast.
• Distributed scientific tools
• Python! Sometimes.
• Cutting edge technology

8. Setting up the workflow
• Spark journey
– Hadoop server: 1.2
– EMR: 1.3
– EMR: 1.4
– EMR: 1.5! Jupyter Notebook running Scala!

9. How do I use it?
• EMR!
• spark-submit on the cluster
• Can add script as a step to cluster launch

10. Let’s set the stage…
• Understanding user behavior
• How do I extract, explore, and model a subset
of our data using Spark?

11. Data
{"a": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4
Safari/600.4.10",
"c": "US",
"nk": 0,
"tz": "America/Los_Angeles",
"g": "1HfTjh8",
"h": "1HfTjh7",
"u": "http://www.nytimes.com/2015/03/22/opinion/sunday/why-
health-care-tech-is-still-so-bad.html?smid=tw-share",
"t": 1427288425,
"cy": "Seattle"}

12. Data processing
• Problem: I want to retrieve NYT decodes
• Solution: well, there are two…

13. Data processing

14. Data processing

15. Data processing
• SparkSQL: 8 minutes
• Pure Spark: 4 minutes!!!

16. Data processing

17. Exploratory data analysis
• Problem: what’s going on with my decodes?
• Solution: DataFrames!
– Similar to Pandas: describe, drop, fill, aggregate
functions
– You can actually convert to a Pandas DataFrame!

18. Exploratory data analysis
• Get a sense of what’s going on in the data
• Look at distributions, frequencies
• Mostly categorical data here

19. Topic modeling
• Problem: we have so many links but no way to
classify them into certain kinds of content
• Solution: LDA (latent Dirichlet allocation)
– Sort of – compare to other solutions

20. Topic modeling
• Oh, the JVM…
– LDA only in Scala
• Scala jar file
• Store script in S3

21. Topic modeling
• LDA in Spark
– Generative model
– Several different methods
– Term frequency vector as input
• “Note: LDA is a new feature with some missing
functionality...”

22. Topic modeling

23. Topic modeling
• Term frequency vector
TERM
DOCUMENT
python data hot dogs baseball zoo
doc_1 1 3 0 0 0
doc_2 0 0 4 1 0
doc_3 4 0 0 0 5

24. Topic modeling

25. Topic modeling

26. Topic modeling
• Why not??
– Means to an end
– Current large scale scraping inability

27. Trend Detection
• Eventually realtime with Spark Streaming

28. Architecture
• Right now: not in production
– Buy-in
• Streaming applications for parts of the app
• Python or Scala?
– Scala by force (LDA, GraphX)

29. Some issues
• Hadoop servers
• JVM
• gzip
• 1.4
• Resource allocation
• Really only got it to this stage very recently

30. Where to go next?
• Spark in production!
• Use for various parts of our app
• Use for R&D and prototyping purposes, with
the potential to expand into the product

31. Current/future projects
• Trend detection
• Device prediction
• User affinities
– GraphX!
• A/B testing

32. Resources
• spark.apache.org - documentation
• Databricks blog
• Cloudera blog

33. Thanks!!
@sarah_guido