Danielle Jabin is a data engineer at Spotify who works on A/B testing infrastructure. She describes Spotify's big data landscape, which includes over 40 million active users generating 1.5 TB of compressed data per day. Spotify collects this user data using Kafka for high-volume data collection, processes it using Hadoop on a large cluster, and stores aggregates in databases like PostgreSQL and Cassandra for analytics and visualization.

1. June 12, 2014
Danielle Jabin
Data Engineer, A/B Testing
Data at Spotify

2. I’m Danielle Jabin
• Data Engineer in the Stockholm office
• A/B testing infrastructure
• California born & raised
• If I can survive a Swedish winter, so can you!
• Studied Computer Science, Statistics, and Real Estate
through the M&T program at the University of
Pennsylvania

3. 3
Over 40 million active
users
As of June 9, 2014

4. 4
Access to more than 20
million songs
As of June 9, 2014

5. Big Data
• 40 million Monthly Active Users
• 20+ million tracks
• 1.5 TB of compressed data from users per day
• 64 TB of data generated in Hadoop each day (including
replication factor of 3)
As of June 9, 2014

6. 6
So how much data is that?

7. Let’s compare: 64 TB
• 293, 203, 072 books (200 pages or 240,000
characters)
• 16,777,216 MP3 files (with 4MB average file size)
• 22,369,600 images (with 3MB average file size)

8. 8
That’s a lot of selfies

9. 9
How do we use this data?

10. Use Cases
• Reporting
• Business Analytics
• Operational Analytics
• Product Features

11. Reporting
• Reporting to labels, licensors, partners, and advertisers
• We support our partners

12. Business Analytics
• Analyzing growth, user behavior, sign-up funnels, etc
• Company KPIs
• NPS analysis

13. Operational Metrics
• Root cause analysis
• Latency analysis
• Better capacity planning (servers, people, bandwidth)

14. Product Features
• Discover and Radio
• Top lists
• Personalized recommendations
• A/B Testing

15. 15
How do we collect this
data?

16. The three pillars of our Data Infrastructure:
Kafka
Collection
Hadoop
Processing
Databases
Analytics/Visualization

17. This is Dave. Data Engineer at
Spotify by day…

18. …chiptune DJ Demoscene Time
Machine by night.

19. Let’s listen to Dave’s song

20. Kafka
• High volume pub-sub
system
• “Producers publish messages to
Kafka topics, and consumers
subscribe to these topics and
consume the messages.”

21. Kafka
• Robust and scalable solution for collection of logs
• Fast data transfer
• Low CPU overhead
• Built-in partitioning, replication, and fault-tolerance
• Consumers can pull data at different rates
• Able to handle extremely high volumes

22. Other people listened too!

23. Hadoop
• Process and store massive amounts of unstructured data
across a distributed cluster
• One cluster with 37 nodes to 690 nodes today
• 28 PB of storage
• The largest Hadoop cluster in Europe

24. Hadoop
• Entering the land of optimizations
• Data retention policy
• Move to JVM-based languages
• MapReduce languages
• Moving to Crunch, JVM-based, for speed and scalability
• Python with Hadoop Streaming, Java, Hive, PIG, Scala
• Sprunch: Crunch wrapper for Scala, open sourced by Spotify
• Spotify open-sourced scheduler, Luigi, written in Python
• Simple and easy way to chain jobs

25. What if we want to know more?
vs

26. Databases
• Aggregates from Hadoop put into PostgreSQL or
Cassandra
• Sqoop
• Core data can be used and manipulated for various needs
• Ad hoc queries
• Dashboards

27. Databases
• Aggregates from Hadoop put into PostgreSQL or
Cassandra
• Sqoop
• Ad hoc queries
• Dashboards

28. Databases
• Aggregates from Hadoop put into PostgreSQL or
Cassandra
• Sqoop
• Ad hoc queries
• Dashboards

29. Questions?

30. A/B testing questions? Find me!
Contr
ol
vs

31. Thank you!