Quality Built In @ Spotify

Andrii Dzynia
Andrii DzyniaSoftware Engineer at Spotify
December 6, 2014 
Andrii Dzynia 
Road Manager to Continuous Delivery 
@adzynia 
Quality Built In
2 
Spotify brings you the right music for 
every moment! 
Started in 2006 (in Sweden) 
Now 1500+ employees, 500+ engineers 
Over 30 million songs available 
Over 20,000 songs added every day 
5 development centres across the globe
What is Quality? 
3
4 
Quality is a “state” …. 
when expectations matches the reality. 
Not just software defects, but stakeholders expectations as well
Why Quality is Important? 
5
Cost of Bug/Change 
6
What is a way to build high 
quality product? 
7
8 Do not release until it’s perfect
9 
No one is perfect.
Example 
Implementing internal back office system
11 
Step 1 - Prototyping phase 
Technical spike investigation 
Hacking different Back-End solutions 
Ad-hoc design discussions 
Stub implementation of Front End 
Getting early user’s feedback on Front End
12 WebApp 
with 
stubs 
Sketches 
on 
whiteboard 
Whatever works 
to get end-user 
understanding
13 
Step 2 - Setting delivery pipeline 
Add more unit tests 
Add more integration tests 
Add more functional tests 
Automate deployment configuration 
Add dashboards and system monitoring 
Clarifying End User Needs
14 
http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment
15 
Step 3 - Connecting the dots 
Connect back-end and front-end parts 
into 
End-2-End system 
Get user’s feedback and iterate
16 
Keep releasing small, usable, incremental pieces
17 
What is under the hood ?
18
Organic Structure 
19 
@joakimsunden Scaled Agile @ Spotify
20 
Test Automation Pyramid
21 
How to avoid Ice-Cream effect?
22 
Be strict with this rule.
Testable micro-services architecture 23 
Unit Tests 
Component Tests 
Functional 
Tests 
Integration Tests 
System/End-2-End Tests 
Unit Tests 
Component Tests 
Functional 
Tests 
JS Unit Tests 
JS Component Tests 
UI 
Functional 
Tests
24 
Unit Tests 
Check logic of minimal code snippet 
with no or mocked dependencies 
https://github.com/mockito/mockito 
https://github.com/junit-team/junit
25 
Component Tests 
Very much like unit tests but emulating 
external components 
e.g. cassandra database 
https://github.com/jsevellec/cassandra-unit
26 
Fake back-end for client apps 
Client App 
Fake Back-End Server 
Spotify Back End 
https://github.com/azagniotov/stubby4j 
https://github.com/dreamhead/moco 
http://jsonstub.com
27 
Integration Tests 
Check an integration between components 
You do not need whole system to check the integration 
https://github.com/spotify/helios/blob/master/docs/testing_framework.md
28 
Create Test Environment ‘on the fly’ 
@docker containers 
https://www.docker.com
29 
Functional Tests 
Check services API behaviour 
or 
End user use cases 
http://hc.apache.org 
http://seleniumhq.org
30 
Model Based Testing 
http://graphwalker.org
Quality Built In @ Spotify
32 
Code Review 
& 
Human Testing 
on 
Pull Request
Delivery Pipeline 
33
34 
Was not covered during this talk … 
Load Testing 
Gradual Rollouts 
A/B Testing 
Feature Toggles 
Monitoring/Alerting 
Information Radiators 
… but stay tuned
35 
Test Engineering Roles and Responsibilities 
Technical Test Engineer Test Engineer 
! = Test Automator ! = Manual Tester 
~ Software Engineer in Test 
~ Context Driven Tester
36 
Technical Test Engineer 
Work as a software developer 
Advocate testability of the product 
Argue on software design with software engineers 
Help with building new/injecting existed development tools
37 
Development Productivity Tools 
e.g. cli control over amazon cloud http://dashing.io 
Docker orchestration 
https://github.com/spotify/helios 
Cassandra Unit 
https://github.com/jsevellec/cassandra-unit 
Spoticloud 
Jenkins job-dsl-plugin 
https://github.com/mikaellanger/job-dsl-plugin 
Dashboards
38 
Test Engineer 
Knows business domain 
Focused on exploring the product 
Free to use any programming language to test specific use case 
Free to use any programming language to automate his work
39 
Mind Map as a Tool 
https://www.mindmup.com 
Product tree 
Scenarios 
Playbooks 
Checklists 
Session notes
Quality Engineers 
40 
@jamesmarcusbach calls us Test Jumpers 
TEST JUMPERS: ONE VISION OF AGILE TESTING 
http://www.satisfice.com/blog/archives/1372 
“RESPONSIBLE TESTER” 
http://www.satisfice.com/blog/archives/1364 
OMEGA TESTER 
http://www.satisfice.com/articles/omega_tester.pdf
41 
Any other ways 
to improve 
quality?
42 
Break Uncertainty 
test ideas during healthy 
discussions 
test requirements via end 
users collaboration 
discuss system design and 
conner cases earlier in the 
planing/design meetings 
define definition of done
43 
Use Google Docs collaboration for finding group consensus 
But have responsible person to figure out what consensus means in each case
Social Programming 
44 
write “checks” during implementation 
shape your thoughts via pair discussions 
peer review before merging to master
45 
Not mentioned Engineering Practices 
User Stories 
Planning meeting 
Standup meeting 
Open workspace 
TDD 
Refactoring 
etc…… 
http://www.extremeprogramming.org/rules.html
Employee Early Builds Testing 
46
47 
Hackathons 
One week to do whatever 
you want 
Do it alone or with a team 
Crazy ideas comes out
48 
QA engineer at Spotify 
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/
How to check you are on 
a right path ? 
49
That is easy to spot 50
51 Ready for a journey? 
Make Quality explicit 
Find Quality promoters 
Define your way of Quality 
improvements 
Set right Quality constraints 
Share results as early as possible 
Keep looking further quality improvements 
Probably you will never end :)
52 
@adzynia 
adzynia@spotify.com 
Spotify Engineering Culture 
part 1 
part 2
and … 
Stop saying QA 
when you mean Testing 
Want to join the band? 
Check out http://www.spotify.com/jobs or @Spotifyjobs for more information
1 of 53

More Related Content

What's hot(20)

functional testing functional testing
functional testing
bharathanche7.2K views
Agile QA presentationAgile QA presentation
Agile QA presentation
Carl Bruiners15.4K views
API TESTINGAPI TESTING
API TESTING
Sijan Bhandari590 views
Automation Testing With AppiumAutomation Testing With Appium
Automation Testing With Appium
Knoldus Inc.5.2K views
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile Testing
Raymond Adrian (Rad) Butalid373 views
Test automationTest automation
Test automation
Xavier Yin1.4K views
API TestingAPI Testing
API Testing
Bikash Sharma7.8K views
Tracking and improving software quality with SonarQubeTracking and improving software quality with SonarQube
Tracking and improving software quality with SonarQube
Patroklos Papapetrou (Pat)19.4K views
Software Testing 101Software Testing 101
Software Testing 101
QA Hannah4.7K views
Introduction & Manual TestingIntroduction & Manual Testing
Introduction & Manual Testing
VenkateswaraRao Siddabathula6.4K views
Getting started with appiumGetting started with appium
Getting started with appium
Pratik Patel615 views
BDD with CucumberBDD with Cucumber
BDD with Cucumber
Knoldus Inc.15.8K views

Similar to Quality Built In @ Spotify(20)

Testing practicies not only in scalaTesting practicies not only in scala
Testing practicies not only in scala
Paweł Panasewicz190 views
DevOps and Build AutomationDevOps and Build Automation
DevOps and Build Automation
Heiswayi Nrird847 views
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
Araf Karsh Hamid1.2K views
Gherkin /BDD introGherkin /BDD intro
Gherkin /BDD intro
Ryan Thomas Hewitt★CSM★CSPO★ITIL3.9K views

Quality Built In @ Spotify

  • 1. December 6, 2014 Andrii Dzynia Road Manager to Continuous Delivery @adzynia Quality Built In
  • 2. 2 Spotify brings you the right music for every moment! Started in 2006 (in Sweden) Now 1500+ employees, 500+ engineers Over 30 million songs available Over 20,000 songs added every day 5 development centres across the globe
  • 4. 4 Quality is a “state” …. when expectations matches the reality. Not just software defects, but stakeholders expectations as well
  • 5. Why Quality is Important? 5
  • 7. What is a way to build high quality product? 7
  • 8. 8 Do not release until it’s perfect
  • 9. 9 No one is perfect.
  • 10. Example Implementing internal back office system
  • 11. 11 Step 1 - Prototyping phase Technical spike investigation Hacking different Back-End solutions Ad-hoc design discussions Stub implementation of Front End Getting early user’s feedback on Front End
  • 12. 12 WebApp with stubs Sketches on whiteboard Whatever works to get end-user understanding
  • 13. 13 Step 2 - Setting delivery pipeline Add more unit tests Add more integration tests Add more functional tests Automate deployment configuration Add dashboards and system monitoring Clarifying End User Needs
  • 15. 15 Step 3 - Connecting the dots Connect back-end and front-end parts into End-2-End system Get user’s feedback and iterate
  • 16. 16 Keep releasing small, usable, incremental pieces
  • 17. 17 What is under the hood ?
  • 18. 18
  • 19. Organic Structure 19 @joakimsunden Scaled Agile @ Spotify
  • 21. 21 How to avoid Ice-Cream effect?
  • 22. 22 Be strict with this rule.
  • 23. Testable micro-services architecture 23 Unit Tests Component Tests Functional Tests Integration Tests System/End-2-End Tests Unit Tests Component Tests Functional Tests JS Unit Tests JS Component Tests UI Functional Tests
  • 24. 24 Unit Tests Check logic of minimal code snippet with no or mocked dependencies https://github.com/mockito/mockito https://github.com/junit-team/junit
  • 25. 25 Component Tests Very much like unit tests but emulating external components e.g. cassandra database https://github.com/jsevellec/cassandra-unit
  • 26. 26 Fake back-end for client apps Client App Fake Back-End Server Spotify Back End https://github.com/azagniotov/stubby4j https://github.com/dreamhead/moco http://jsonstub.com
  • 27. 27 Integration Tests Check an integration between components You do not need whole system to check the integration https://github.com/spotify/helios/blob/master/docs/testing_framework.md
  • 28. 28 Create Test Environment ‘on the fly’ @docker containers https://www.docker.com
  • 29. 29 Functional Tests Check services API behaviour or End user use cases http://hc.apache.org http://seleniumhq.org
  • 30. 30 Model Based Testing http://graphwalker.org
  • 32. 32 Code Review & Human Testing on Pull Request
  • 34. 34 Was not covered during this talk … Load Testing Gradual Rollouts A/B Testing Feature Toggles Monitoring/Alerting Information Radiators … but stay tuned
  • 35. 35 Test Engineering Roles and Responsibilities Technical Test Engineer Test Engineer ! = Test Automator ! = Manual Tester ~ Software Engineer in Test ~ Context Driven Tester
  • 36. 36 Technical Test Engineer Work as a software developer Advocate testability of the product Argue on software design with software engineers Help with building new/injecting existed development tools
  • 37. 37 Development Productivity Tools e.g. cli control over amazon cloud http://dashing.io Docker orchestration https://github.com/spotify/helios Cassandra Unit https://github.com/jsevellec/cassandra-unit Spoticloud Jenkins job-dsl-plugin https://github.com/mikaellanger/job-dsl-plugin Dashboards
  • 38. 38 Test Engineer Knows business domain Focused on exploring the product Free to use any programming language to test specific use case Free to use any programming language to automate his work
  • 39. 39 Mind Map as a Tool https://www.mindmup.com Product tree Scenarios Playbooks Checklists Session notes
  • 40. Quality Engineers 40 @jamesmarcusbach calls us Test Jumpers TEST JUMPERS: ONE VISION OF AGILE TESTING http://www.satisfice.com/blog/archives/1372 “RESPONSIBLE TESTER” http://www.satisfice.com/blog/archives/1364 OMEGA TESTER http://www.satisfice.com/articles/omega_tester.pdf
  • 41. 41 Any other ways to improve quality?
  • 42. 42 Break Uncertainty test ideas during healthy discussions test requirements via end users collaboration discuss system design and conner cases earlier in the planing/design meetings define definition of done
  • 43. 43 Use Google Docs collaboration for finding group consensus But have responsible person to figure out what consensus means in each case
  • 44. Social Programming 44 write “checks” during implementation shape your thoughts via pair discussions peer review before merging to master
  • 45. 45 Not mentioned Engineering Practices User Stories Planning meeting Standup meeting Open workspace TDD Refactoring etc…… http://www.extremeprogramming.org/rules.html
  • 46. Employee Early Builds Testing 46
  • 47. 47 Hackathons One week to do whatever you want Do it alone or with a team Crazy ideas comes out
  • 48. 48 QA engineer at Spotify http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/
  • 49. How to check you are on a right path ? 49
  • 50. That is easy to spot 50
  • 51. 51 Ready for a journey? Make Quality explicit Find Quality promoters Define your way of Quality improvements Set right Quality constraints Share results as early as possible Keep looking further quality improvements Probably you will never end :)
  • 52. 52 @adzynia adzynia@spotify.com Spotify Engineering Culture part 1 part 2
  • 53. and … Stop saying QA when you mean Testing Want to join the band? Check out http://www.spotify.com/jobs or @Spotifyjobs for more information