This document summarizes Spotify's approach to continuous delivery and building quality into software development. It discusses that quality involves meeting stakeholder expectations, and the importance of quality to reduce costs from bugs. Spotify emphasizes automated testing from unit to integration levels. They implement a delivery pipeline to continuously integrate and deploy code. Other practices include code reviews, monitoring, and allowing employees time for experiments.

1. December 6, 2014
Andrii Dzynia
Road Manager to Continuous Delivery
@adzynia
Quality Built In

2. 2
Spotify brings you the right music for
every moment!
Started in 2006 (in Sweden)
Now 1500+ employees, 500+ engineers
Over 30 million songs available
Over 20,000 songs added every day
5 development centres across the globe

3. What is Quality?
3

4. 4
Not just software defects, but stakeholders expectations as well
when expectations matches the reality.
Quality is a “state” ….

5. Why Quality is Important?
5

6. Cost of Bug/Change
6

7. What is a way to build high
quality product?
7

8. 8
Do not release until it’s perfect

9. 9
No one is perfect.

10. Implementing internal back office system
Example

11. 11
Step 1 - Prototyping phase
Technical spike investigation
Hacking different Back-End solutions
Ad-hoc design discussions
Stub implementation of Front End
Getting early user’s feedback on Front End

12. 12
WebApp
with
stubs
Whatever works
to get end-user
understanding
Sketches
on
whiteboard

13. 13
Step 2 - Setting delivery pipeline
Add more unit tests
Add more integration tests
Add more functional tests
Automate deployment configuration
Add dashboards and system monitoring
Clarifying End User Needs

14. 14
http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment

15. 15
Step 3 - Connecting the dots
Connect back-end and front-end parts
into
End-2-End system
Get user’s feedback and iterate

16. Keep releasing small, usable, incremental pieces
16

17. 17
What is under the hood ?

18. 18

19. Organic Structure
19
Scaled Agile @ Spotify@joakimsunden

20. 20
Test Automation Pyramid

21. 21
How to avoid Ice-Cream effect?

22. 22
Be strict with this rule.

23. Testable micro-services architecture
23
Unit Tests
Component Tests
Functional
Tests
System/End-2-End Tests
Integration Tests
Unit Tests
Component Tests
Functional
Tests
JS Unit Tests
JS Component Tests
UI
Functional
Tests

24. 24
Unit Tests
Check logic of minimal code snippet
with no or mocked dependencies
https://github.com/mockito/mockito
https://github.com/junit-team/junit

25. 25
https://github.com/jsevellec/cassandra-unit
Very much like unit tests but emulating
external components
e.g. cassandra database
Component Tests

26. 26
Client App
Fake Back-End Server
Spotify Back End
Fake back-end for client apps
http://jsonstub.com
https://github.com/azagniotov/stubby4j
https://github.com/dreamhead/moco

27. 27
Integration Tests
Check an integration between components
You do not need whole system to check the integration
https://github.com/spotify/helios/blob/master/docs/testing_framework.md

28. 28
@docker containers
Create Test Environment ‘on the fly’
https://www.docker.com

29. 29
Check services API behaviour
or
End user use cases
Functional Tests
http://seleniumhq.org
http://hc.apache.org

30. 30
http://graphwalker.org
Model Based Testing

32. 32
Code Review
&
Human Testing
on
Pull Request

33. Delivery Pipeline
33

34. 34
Was not covered during this talk …
Load Testing
Gradual Rollouts
A/B Testing
Feature Toggles
Monitoring/Alerting
Information Radiators
… but stay tuned

35. 35
Technical Test Engineer Test Engineer
! = Manual Tester! = Test Automator
~ Software Engineer in Test
Test Engineering Roles and Responsibilities
~ Context Driven Tester

36. 36
Technical Test Engineer
Work as a software developer
Advocate testability of the product
Argue on software design with software engineers
Help with building new/injecting existed development tools

37. 37
https://github.com/jsevellec/cassandra-unit
Cassandra Unit
https://github.com/mikaellanger/job-dsl-plugin
Jenkins job-dsl-plugin
https://github.com/spotify/helios
Docker orchestration
Spoticloud
e.g. cli control over amazon cloud http://dashing.io
Dashboards
Development Productivity Tools

38. 38
Knows business domain
Focused on exploring the product
Free to use any programming language to test specific use case
Free to use any programming language to automate his work
Test Engineer

39. 39
Mind Map as a Tool
https://www.mindmup.com
Product tree
Scenarios
Playbooks
Checklists
Session notes

40. Quality Engineers
40
http://www.satisfice.com/blog/archives/1372
TEST JUMPERS: ONE VISION OF AGILE TESTING
http://www.satisfice.com/blog/archives/1364
“RESPONSIBLE TESTER”
http://www.satisfice.com/articles/omega_tester.pdf
OMEGA TESTER
@jamesmarcusbach calls us Test Jumpers

41. 41
Any other ways
to improve
quality?

42. 42
test ideas during healthy
discussions
test requirements via end
users collaboration
discuss system design and
conner cases earlier in the
planing/design meetings
define definition of done
Break Uncertainty

43. 43
But have responsible person to figure out what consensus means in each case
Use Google Docs collaboration for finding group consensus

44. Social Programming
44
write “checks” during implementation
shape your thoughts via pair discussions
peer review before merging to master

45. 45
User Stories
Planning meeting
Standup meeting
Open workspace
TDD
Refactoring
etc……
Not mentioned Engineering Practices
http://www.extremeprogramming.org/rules.html

46. Employee Early Builds Testing
46

47. 47
One week to do whatever
you want
Do it alone or with a team
Crazy ideas comes out
Hackathons

48. 48
QA engineer at Spotify
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

49. How to check you are on
a right path ?
49

50. That is easy to spot 50

51. 51
Ready for a journey?
Make Quality explicit
Find Quality promoters
Define your way of Quality
improvements
Set right Quality constraints
Share results as early as possible
Keep looking further quality improvements
Probably you will never end :)

52. 52
@adzynia
adzynia@spotify.com
Spotify Engineering Culture
part 1
part 2

53. and …
Stop saying QA
when you mean Testing
Check out http://www.spotify.com/jobs or @Spotifyjobs for more information
Want to join the band?