Check out a presentation from Sarah Guido, data scientist at Bitly, exploring Apache Spark and data science at scale.

1. Data Science at Scale:
Using Apache Spark for Data Science
at Bitly
Sarah Guido
Data Day Seattle 2015

2. Overview
• About me/Bitly
• Spark overview
• Using Spark for data science
• When it works, it’s great! When it works…

3. About me
• Data scientist at Bitly
• NYC Python/PyGotham co-organizer
• O’Reilly Media author
• @sarah_guido

4. About this talk
• This talk is:
– Description of my workflow
– Exploration of within-Spark tools
• This talk is not:
– In-depth exploration of algorithms
– Building new tools on top of Spark
– Any sort of ground truth for how you should be
using Spark

5. A bit of background
• Need for big data analysis tools
• MapReduce for exploratory data analysis == 
• Iterate/prototype quickly
• Overall goal: understand how people use not
only our app, but the Internet!

6. Bitly data!
• Legit big data
• 1 hour of decodes is 10 GB
• 1 day is 240 GB
• 1 month is ~7 TB

7. Why Spark?
• Fast. Really fast.
• Distributed scientific tools
• Python! Sometimes.
• Cutting edge technology

8. Setting up the workflow
• Spark journey
– Hadoop server: 1.2
– EMR: 1.3
– EMR: 1.4
– EMR: 1.5! Jupyter Notebook running Scala!

9. How do I use it?
• EMR!
• spark-submit on the cluster
• Can add script as a step to cluster launch

10. Let’s set the stage…
• Understanding user behavior
• How do I extract, explore, and model a subset
of our data using Spark?

11. Data
{"a": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4
Safari/600.4.10",
"c": "US",
"nk": 0,
"tz": "America/Los_Angeles",
"g": "1HfTjh8",
"h": "1HfTjh7",
"u": "http://www.nytimes.com/2015/03/22/opinion/sunday/why-
health-care-tech-is-still-so-bad.html?smid=tw-share",
"t": 1427288425,
"cy": "Seattle"}

12. Data processing
• Problem: I want to retrieve NYT decodes
• Solution: well, there are two…

13. Data processing

14. Data processing

15. Data processing
• SparkSQL: 8 minutes
• Pure Spark: 4 minutes!!!

16. Data processing

17. Exploratory data analysis
• Problem: what’s going on with my decodes?
• Solution: DataFrames!
– Similar to Pandas: describe, drop, fill, aggregate
functions
– You can actually convert to a Pandas DataFrame!

18. Exploratory data analysis
• Get a sense of what’s going on in the data
• Look at distributions, frequencies
• Mostly categorical data here

19. Topic modeling
• Problem: we have so many links but no way to
classify them into certain kinds of content
• Solution: LDA (latent Dirichlet allocation)
– Sort of – compare to other solutions

20. Topic modeling
• Oh, the JVM…
– LDA only in Scala
• Scala jar file
• Store script in S3

21. Topic modeling
• LDA in Spark
– Generative model
– Several different methods
– Term frequency vector as input
• “Note: LDA is a new feature with some missing
functionality...”

22. Topic modeling

23. Topic modeling
• Term frequency vector
TERM
DOCUMENT
python data hot dogs baseball zoo
doc_1 1 3 0 0 0
doc_2 0 0 4 1 0
doc_3 4 0 0 0 5

24. Topic modeling

25. Topic modeling

26. Topic modeling
• Why not??
– Means to an end
– Current large scale scraping inability

27. Trend Detection
• Eventually realtime with Spark Streaming

28. Architecture
• Right now: not in production
– Buy-in
• Streaming applications for parts of the app
• Python or Scala?
– Scala by force (LDA, GraphX)

29. Some issues
• Hadoop servers
• JVM
• gzip
• 1.4
• Resource allocation
• Really only got it to this stage very recently

30. Where to go next?
• Spark in production!
• Use for various parts of our app
• Use for R&D and prototyping purposes, with
the potential to expand into the product

31. Current/future projects
• Trend detection
• Device prediction
• User affinities
– GraphX!
• A/B testing

32. Resources
• spark.apache.org - documentation
• Databricks blog
• Cloudera blog

33. Thanks!!
@sarah_guido