This document discusses using a data-driven approach to product development. It recommends incorporating data into every step of development by: 1) Analyzing data to gain insights on what is useful for users; 2) Measuring and testing solutions through A/B testing to determine if an approach solves the intended problem; and 3) Continually collecting user data and feedback to inform the process. The document advocates for cross-functional squads and using data insights to drive ideas rather than assumptions. The goal is to develop features and experiences that truly benefit users based on empirical evidence.

1. Datain
development
Oscar Carlsson
Data Scientist @ Spotify

4. Move fast!
No decision making bottlenecks
Every squad should be autonomous
No dependencies

5. Product Owner
QA
Client dev(s)
Backend dev(s)
Designer
DataAnalyst

6. Playlist Your Music
Audio playback
Göteborg

7. Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it

8. Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
“This seems
reasonable to do”
“I would really
like this feature”

9. Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
What ifthe idea
isn’t useful to users?
Nobody uses this.
Who came up
with this idea?

10. Are we representative of our users?
totally fictional graph to illustrate that employees who use the product is probably not representative ofthe normal users!
Ratioofusers
0
0,225
0,45
0,675
0,9
Engagement score
0 1 2 3 4 5 6 7
Normal users Employees

11. Data to the rescue!

12. 1. Analyse the data to
get insights on what is
useful for real users

13. A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights

14. A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Probably something that is useful

15. A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Probably something that is useful This particular solution works.
But is it the best?

16. A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Are we sure what we are
working on solving the problem we saw?

17. A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype
One platform only?
Think it Code it Release itInsights
Code it

18. How to test this?
A/B testing!

19. A/B testing
Statistical hypothesis testing
Simple randomized experiment with >= 2 variants (A, B)

20. A/B testing
Control A B
“The shuffle button test”
Hypothesis: From insights Xwe believe
redesigning the shuffle button will
increase number of users who click on it

21. A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Fictional numbers!

22. A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clicksperuser/day
0
1
2
3
4
5
Control A B
Fictional numbers!

23. A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clicksperuser/day
0
1
2
3
4
5
Control A B
Retention
Ratioofusersthatclickagain
0
3,75
7,5
11,25
15
Control A B
Fictional numbers!

24. 2. Measure and test if
what you are doing is
solving what you hoped to

25. Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype Code it
Maybe do something else

26. Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype Code it
Data Data Data

27. All of this requires collection of
data
and users to test on

28. Data
Measures from the clients and services
User surveys
User interviews

29. Measures from the clients and services
> 30 TB per day
Event logs
2000 machine Hadoop cluster (moving to GCP)

30. Playlist and Your
Music squad

32. More than
2 billion
playlists

33. Less users
create playlist
on iOS
Fictional insights that could be presented to the Playlist squad
Some users don’t
invest time in
creating playlists
Revisit
playlist flow
on iOS
Pre curated
playlists

34. Audio playback
squad

35. 30 million songs
to
75 million users
in
59 countries

37. Swedish data plans are
enormous compared to
countries X,Y,Z
Fictional insights that could be presented to theAudio playback squad
Some users only
have internet at home
but want to
stream everywhere
Offline, caching?
Improve
Offline
Experience

38. Summary
• Incorporate data in all steps of development and in organisation
• Cross functional squads - feature teams
• Let Data/Insights drive ideas
• A/B tests to evaluate solutions

39. Thanks for listening!
Oscar Carlsson
lad@spotify.com
@Oscarlsson
Oscarlsson1