Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Datain
development
Oscar Carlsson
Data Scientist @ Spotify
Move fast!
No decision making bottlenecks
Every squad should be autonomous
No dependencies
Product Owner
QA
Client dev(s)
Backend dev(s)
Designer
DataAnalyst
Playlist Your Music
Audio playback
Göteborg
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
“This seems
reasonable to ...
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
What ifthe idea
isn’t usef...
Are we representative of our users?
totally fictional graph to illustrate that employees who use the product is probably n...
Data to the rescue!
1. Analyse the data to
get insights on what is
useful for real users
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Prob...
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Prob...
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Are ...
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPro...
How to test this?
A/B testing!
A/B testing
Statistical hypothesis testing
Simple randomized experiment with >= 2 variants (A, B)
A/B testing
Control A B
“The shuffle button test”
Hypothesis: From insights Xwe believe
redesigning the shuffle button wil...
A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Fictional numbers!
A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clicksperuser/day
0
1
2
3...
A/B testing Reach
Shareofuserswhopress
shuffle(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clicksperuser/day
0
1
2
3...
2. Measure and test if
what you are doing is
solving what you hoped to
Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototy...
Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototy...
All of this requires collection of
data
and users to test on
Data
Measures from the clients and services
User surveys
User interviews
Measures from the clients and services
> 30 TB per day
Event logs
2000 machine Hadoop cluster (moving to GCP)
Playlist and Your
Music squad
More than
2 billion
playlists
Less users
create playlist
on iOS
Fictional insights that could be presented to the Playlist squad
Some users don’t
invest...
Audio playback
squad
30 million songs
to
75 million users
in
59 countries
Swedish data plans are
enormous compared to
countries X,Y,Z
Fictional insights that could be presented to theAudio playbac...
Summary
• Incorporate data in all steps of development and in organisation
• Cross functional squads - feature teams
• Let...
Thanks for listening!
Oscar Carlsson
lad@spotify.com
@Oscarlsson
Oscarlsson1
Data in development @ Spotify
Data in development @ Spotify
Data in development @ Spotify
Data in development @ Spotify
Upcoming SlideShare
Loading in …5
×

Data in development @ Spotify

621 views

Published on

Talk given at Sigma Gothenburg AWS 2 May 2016.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Data in development @ Spotify

  1. 1. Datain development Oscar Carlsson Data Scientist @ Spotify
  2. 2. Move fast! No decision making bottlenecks Every squad should be autonomous No dependencies
  3. 3. Product Owner QA Client dev(s) Backend dev(s) Designer DataAnalyst
  4. 4. Playlist Your Music Audio playback Göteborg
  5. 5. Traditional development illustration courtesy of Dan McKinley, Etsy Think it Code it Release it
  6. 6. Traditional development illustration courtesy of Dan McKinley, Etsy Think it Code it Release it “This seems reasonable to do” “I would really like this feature”
  7. 7. Traditional development illustration courtesy of Dan McKinley, Etsy Think it Code it Release it What ifthe idea isn’t useful to users? Nobody uses this. Who came up with this idea?
  8. 8. Are we representative of our users? totally fictional graph to illustrate that employees who use the product is probably not representative ofthe normal users! Ratioofusers 0 0,225 0,45 0,675 0,9 Engagement score 0 1 2 3 4 5 6 7 Normal users Employees
  9. 9. Data to the rescue!
  10. 10. 1. Analyse the data to get insights on what is useful for real users
  11. 11. A little more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Code it Release itInsights
  12. 12. A little more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Code it Release itInsights Probably something that is useful
  13. 13. A little more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Code it Release itInsights Probably something that is useful This particular solution works. But is it the best?
  14. 14. A little more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Code it Release itInsights Are we sure what we are working on solving the problem we saw?
  15. 15. A little more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Release itInsights Test TestPrototype One platform only? Think it Code it Release itInsights Code it
  16. 16. How to test this? A/B testing!
  17. 17. A/B testing Statistical hypothesis testing Simple randomized experiment with >= 2 variants (A, B)
  18. 18. A/B testing Control A B “The shuffle button test” Hypothesis: From insights Xwe believe redesigning the shuffle button will increase number of users who click on it
  19. 19. A/B testing Reach Shareofuserswhopress shuffle(%) 0 10 20 30 40 50 Control A B Control A B Fictional numbers!
  20. 20. A/B testing Reach Shareofuserswhopress shuffle(%) 0 10 20 30 40 50 Control A B Control A B Depth Clicksperuser/day 0 1 2 3 4 5 Control A B Fictional numbers!
  21. 21. A/B testing Reach Shareofuserswhopress shuffle(%) 0 10 20 30 40 50 Control A B Control A B Depth Clicksperuser/day 0 1 2 3 4 5 Control A B Retention Ratioofusersthatclickagain 0 3,75 7,5 11,25 15 Control A B Fictional numbers!
  22. 22. 2. Measure and test if what you are doing is solving what you hoped to
  23. 23. Even more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Release itInsights Test TestPrototype Code it Maybe do something else
  24. 24. Even more data-driven development illustration courtesy of Dan McKinley, Etsy Think it Release itInsights Test TestPrototype Code it Data Data Data
  25. 25. All of this requires collection of data and users to test on
  26. 26. Data Measures from the clients and services User surveys User interviews
  27. 27. Measures from the clients and services > 30 TB per day Event logs 2000 machine Hadoop cluster (moving to GCP)
  28. 28. Playlist and Your Music squad
  29. 29. More than 2 billion playlists
  30. 30. Less users create playlist on iOS Fictional insights that could be presented to the Playlist squad Some users don’t invest time in creating playlists Revisit playlist flow on iOS Pre curated playlists
  31. 31. Audio playback squad
  32. 32. 30 million songs to 75 million users in 59 countries
  33. 33. Swedish data plans are enormous compared to countries X,Y,Z Fictional insights that could be presented to theAudio playback squad Some users only have internet at home but want to stream everywhere Offline, caching? Improve Offline Experience
  34. 34. Summary • Incorporate data in all steps of development and in organisation • Cross functional squads - feature teams • Let Data/Insights drive ideas • A/B tests to evaluate solutions
  35. 35. Thanks for listening! Oscar Carlsson lad@spotify.com @Oscarlsson Oscarlsson1

×