December 6, 2014 Andrii Dzynia
Road Manager to Continuous Delivery @adzynia Quality Built In

2 Spotify brings you the
right music for every moment! Started in 2006 (in Sweden) Now 1500+ employees, 500+ engineers Over 30 million songs available Over 20,000 songs added every day 5 development centres across the globe

What is Quality?
3

4 Quality is a “state”
…. when expectations matches the reality. Not just software defects, but stakeholders expectations as well

Why Quality is Important?
5

Cost of Bug/Change
6

What is a way to
build high quality product? 7

8 Do not release until
it’s perfect

9
No one is perfect.

Example Implementing internal back office
system

11 Step 1 - Prototyping
phase Technical spike investigation Hacking different Back-End solutions Ad-hoc design discussions Stub implementation of Front End Getting early user’s feedback on Front End

12 WebApp with stubs Sketches
on whiteboard Whatever works to get end-user understanding

13 Step 2 - Setting
delivery pipeline Add more unit tests Add more integration tests Add more functional tests Automate deployment configuration Add dashboards and system monitoring Clarifying End User Needs

14
http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment

15 Step 3 - Connecting
the dots Connect back-end and front-end parts into End-2-End system Get user’s feedback and iterate

16 Keep releasing small, usable,
incremental pieces

17 What is under the
hood ?

18

Organic Structure 19 @joakimsunden Scaled
Agile @ Spotify

20
Test Automation Pyramid

21 How to avoid Ice-Cream
effect?

22 Be strict with this
rule.

Testable micro-services architecture 23 Unit
Tests Component Tests Functional Tests Integration Tests System/End-2-End Tests Unit Tests Component Tests Functional Tests JS Unit Tests JS Component Tests UI Functional Tests

24 Unit Tests Check logic
of minimal code snippet with no or mocked dependencies https://github.com/mockito/mockito https://github.com/junit-team/junit

25 Component Tests Very much
like unit tests but emulating external components e.g. cassandra database https://github.com/jsevellec/cassandra-unit

26 Fake back-end for client
apps Client App Fake Back-End Server Spotify Back End https://github.com/azagniotov/stubby4j https://github.com/dreamhead/moco http://jsonstub.com

27 Integration Tests Check an
integration between components You do not need whole system to check the integration https://github.com/spotify/helios/blob/master/docs/testing_framework.md

28 Create Test Environment ‘on
the fly’ @docker containers https://www.docker.com

29 Functional Tests Check services
API behaviour or End user use cases http://hc.apache.org http://seleniumhq.org

30
Model Based Testing
http://graphwalker.org

32 Code Review & Human
Testing on Pull Request

Delivery Pipeline
33

34 Was not covered during
this talk … Load Testing Gradual Rollouts A/B Testing Feature Toggles Monitoring/Alerting Information Radiators … but stay tuned

35 Test Engineering Roles and
Responsibilities Technical Test Engineer Test Engineer ! = Test Automator ! = Manual Tester ~ Software Engineer in Test ~ Context Driven Tester

36 Technical Test Engineer Work
as a software developer Advocate testability of the product Argue on software design with software engineers Help with building new/injecting existed development tools

37 Development Productivity Tools e.g.
cli control over amazon cloud http://dashing.io Docker orchestration https://github.com/spotify/helios Cassandra Unit https://github.com/jsevellec/cassandra-unit Spoticloud Jenkins job-dsl-plugin https://github.com/mikaellanger/job-dsl-plugin Dashboards

38 Test Engineer Knows business
domain Focused on exploring the product Free to use any programming language to test specific use case Free to use any programming language to automate his work

39 Mind Map as a
Tool https://www.mindmup.com Product tree Scenarios Playbooks Checklists Session notes

Quality Engineers 40 @jamesmarcusbach calls
us Test Jumpers TEST JUMPERS: ONE VISION OF AGILE TESTING http://www.satisfice.com/blog/archives/1372 “RESPONSIBLE TESTER” http://www.satisfice.com/blog/archives/1364 OMEGA TESTER http://www.satisfice.com/articles/omega_tester.pdf

41 Any other ways to
improve quality?

42 Break Uncertainty test ideas
during healthy discussions test requirements via end users collaboration discuss system design and conner cases earlier in the planing/design meetings define definition of done

43 Use Google Docs collaboration
for finding group consensus But have responsible person to figure out what consensus means in each case

Social Programming 44 write “checks”
during implementation shape your thoughts via pair discussions peer review before merging to master

45 Not mentioned Engineering Practices
User Stories Planning meeting Standup meeting Open workspace TDD Refactoring etc…… http://www.extremeprogramming.org/rules.html

Employee Early Builds Testing
46

47 Hackathons One week to
do whatever you want Do it alone or with a team Crazy ideas comes out

48 QA engineer at Spotify
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

How to check you are
on a right path ? 49

That is easy to spot
50

51 Ready for a journey?
Make Quality explicit Find Quality promoters Define your way of Quality improvements Set right Quality constraints Share results as early as possible Keep looking further quality improvements Probably you will never end :)

52 @adzynia adzynia@spotify.com Spotify Engineering
Culture part 1 part 2

and … Stop saying QA
when you mean Testing Want to join the band? Check out http://www.spotify.com/jobs or @Spotifyjobs for more information