SlideShare a Scribd company logo
December 6, 2014 
Andrii Dzynia 
Road Manager to Continuous Delivery 
@adzynia 
Quality Built In
2 
Spotify brings you the right music for 
every moment! 
Started in 2006 (in Sweden) 
Now 1500+ employees, 500+ engineers 
Over 30 million songs available 
Over 20,000 songs added every day 
5 development centres across the globe
What is Quality? 
3
4 
Quality is a “state” …. 
when expectations matches the reality. 
Not just software defects, but stakeholders expectations as well
Why Quality is Important? 
5
Cost of Bug/Change 
6
What is a way to build high 
quality product? 
7
8 Do not release until it’s perfect
9 
No one is perfect.
Example 
Implementing internal back office system
11 
Step 1 - Prototyping phase 
Technical spike investigation 
Hacking different Back-End solutions 
Ad-hoc design discussions 
Stub implementation of Front End 
Getting early user’s feedback on Front End
12 WebApp 
with 
stubs 
Sketches 
on 
whiteboard 
Whatever works 
to get end-user 
understanding
13 
Step 2 - Setting delivery pipeline 
Add more unit tests 
Add more integration tests 
Add more functional tests 
Automate deployment configuration 
Add dashboards and system monitoring 
Clarifying End User Needs
14 
http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment
15 
Step 3 - Connecting the dots 
Connect back-end and front-end parts 
into 
End-2-End system 
Get user’s feedback and iterate
16 
Keep releasing small, usable, incremental pieces
17 
What is under the hood ?
18
Organic Structure 
19 
@joakimsunden Scaled Agile @ Spotify
20 
Test Automation Pyramid
21 
How to avoid Ice-Cream effect?
22 
Be strict with this rule.
Testable micro-services architecture 23 
Unit Tests 
Component Tests 
Functional 
Tests 
Integration Tests 
System/End-2-End Tests 
Unit Tests 
Component Tests 
Functional 
Tests 
JS Unit Tests 
JS Component Tests 
UI 
Functional 
Tests
24 
Unit Tests 
Check logic of minimal code snippet 
with no or mocked dependencies 
https://github.com/mockito/mockito 
https://github.com/junit-team/junit
25 
Component Tests 
Very much like unit tests but emulating 
external components 
e.g. cassandra database 
https://github.com/jsevellec/cassandra-unit
26 
Fake back-end for client apps 
Client App 
Fake Back-End Server 
Spotify Back End 
https://github.com/azagniotov/stubby4j 
https://github.com/dreamhead/moco 
http://jsonstub.com
27 
Integration Tests 
Check an integration between components 
You do not need whole system to check the integration 
https://github.com/spotify/helios/blob/master/docs/testing_framework.md
28 
Create Test Environment ‘on the fly’ 
@docker containers 
https://www.docker.com
29 
Functional Tests 
Check services API behaviour 
or 
End user use cases 
http://hc.apache.org 
http://seleniumhq.org
30 
Model Based Testing 
http://graphwalker.org
Quality Built In @ Spotify
32 
Code Review 
& 
Human Testing 
on 
Pull Request
Delivery Pipeline 
33
34 
Was not covered during this talk … 
Load Testing 
Gradual Rollouts 
A/B Testing 
Feature Toggles 
Monitoring/Alerting 
Information Radiators 
… but stay tuned
35 
Test Engineering Roles and Responsibilities 
Technical Test Engineer Test Engineer 
! = Test Automator ! = Manual Tester 
~ Software Engineer in Test 
~ Context Driven Tester
36 
Technical Test Engineer 
Work as a software developer 
Advocate testability of the product 
Argue on software design with software engineers 
Help with building new/injecting existed development tools
37 
Development Productivity Tools 
e.g. cli control over amazon cloud http://dashing.io 
Docker orchestration 
https://github.com/spotify/helios 
Cassandra Unit 
https://github.com/jsevellec/cassandra-unit 
Spoticloud 
Jenkins job-dsl-plugin 
https://github.com/mikaellanger/job-dsl-plugin 
Dashboards
38 
Test Engineer 
Knows business domain 
Focused on exploring the product 
Free to use any programming language to test specific use case 
Free to use any programming language to automate his work
39 
Mind Map as a Tool 
https://www.mindmup.com 
Product tree 
Scenarios 
Playbooks 
Checklists 
Session notes
Quality Engineers 
40 
@jamesmarcusbach calls us Test Jumpers 
TEST JUMPERS: ONE VISION OF AGILE TESTING 
http://www.satisfice.com/blog/archives/1372 
“RESPONSIBLE TESTER” 
http://www.satisfice.com/blog/archives/1364 
OMEGA TESTER 
http://www.satisfice.com/articles/omega_tester.pdf
41 
Any other ways 
to improve 
quality?
42 
Break Uncertainty 
test ideas during healthy 
discussions 
test requirements via end 
users collaboration 
discuss system design and 
conner cases earlier in the 
planing/design meetings 
define definition of done
43 
Use Google Docs collaboration for finding group consensus 
But have responsible person to figure out what consensus means in each case
Social Programming 
44 
write “checks” during implementation 
shape your thoughts via pair discussions 
peer review before merging to master
45 
Not mentioned Engineering Practices 
User Stories 
Planning meeting 
Standup meeting 
Open workspace 
TDD 
Refactoring 
etc…… 
http://www.extremeprogramming.org/rules.html
Employee Early Builds Testing 
46
47 
Hackathons 
One week to do whatever 
you want 
Do it alone or with a team 
Crazy ideas comes out
48 
QA engineer at Spotify 
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/
How to check you are on 
a right path ? 
49
That is easy to spot 50
51 Ready for a journey? 
Make Quality explicit 
Find Quality promoters 
Define your way of Quality 
improvements 
Set right Quality constraints 
Share results as early as possible 
Keep looking further quality improvements 
Probably you will never end :)
52 
@adzynia 
adzynia@spotify.com 
Spotify Engineering Culture 
part 1 
part 2
and … 
Stop saying QA 
when you mean Testing 
Want to join the band? 
Check out http://www.spotify.com/jobs or @Spotifyjobs for more information

More Related Content

What's hot

QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
Praveen Dosapati (CSM)
 
Azure dev ops
Azure dev opsAzure dev ops
Azure dev ops
Swaminathan Vetri
 
Test automation methodologies
Test automation methodologiesTest automation methodologies
Test automation methodologies
Mesut Günes
 
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
TEST Huddle
 
Agile testing - Testing From Day 1
Agile testing - Testing From Day 1Agile testing - Testing From Day 1
Agile testing - Testing From Day 1
Kaizenko
 
Testing in Agile Projects
Testing in Agile ProjectsTesting in Agile Projects
Testing in Agile Projects
sriks7
 
Introduction to Software Test Automation
Introduction to Software Test AutomationIntroduction to Software Test Automation
Introduction to Software Test Automation
Amr Ali (ISTQB CTAL Full, CSM, ITIL Foundation)
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
Martin Málek
 
Shift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBMShift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBM
IBM UrbanCode Products
 
Continuous Testing.pptx
Continuous Testing.pptxContinuous Testing.pptx
Continuous Testing.pptx
ShripadH1
 
Effective Test Automation in DevOps
Effective Test Automation in DevOpsEffective Test Automation in DevOps
Effective Test Automation in DevOps
Lee Barnes
 
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
Edureka!
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
Naveen Kumar Singh
 
Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)
Leonard Fingerman
 
Automation Testing
Automation TestingAutomation Testing
Automation Testing
Sun Technlogies
 
Continuous testing
Continuous testing Continuous testing
Continuous testing
Dr Ganesh Iyer
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with Agile
Ken McCorkell
 
Agile Testing Process
Agile Testing ProcessAgile Testing Process
Agile Testing Process
Intetics
 
Qa exploratory test charter template
Qa exploratory test charter templateQa exploratory test charter template
Qa exploratory test charter template
Rob Swoboda
 
Test Automation in Agile
Test Automation in AgileTest Automation in Agile
Test Automation in Agile
Agile Testing Alliance
 

What's hot (20)

QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
Azure dev ops
Azure dev opsAzure dev ops
Azure dev ops
 
Test automation methodologies
Test automation methodologiesTest automation methodologies
Test automation methodologies
 
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
 
Agile testing - Testing From Day 1
Agile testing - Testing From Day 1Agile testing - Testing From Day 1
Agile testing - Testing From Day 1
 
Testing in Agile Projects
Testing in Agile ProjectsTesting in Agile Projects
Testing in Agile Projects
 
Introduction to Software Test Automation
Introduction to Software Test AutomationIntroduction to Software Test Automation
Introduction to Software Test Automation
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
 
Shift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBMShift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBM
 
Continuous Testing.pptx
Continuous Testing.pptxContinuous Testing.pptx
Continuous Testing.pptx
 
Effective Test Automation in DevOps
Effective Test Automation in DevOpsEffective Test Automation in DevOps
Effective Test Automation in DevOps
 
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
 
Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)
 
Automation Testing
Automation TestingAutomation Testing
Automation Testing
 
Continuous testing
Continuous testing Continuous testing
Continuous testing
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with Agile
 
Agile Testing Process
Agile Testing ProcessAgile Testing Process
Agile Testing Process
 
Qa exploratory test charter template
Qa exploratory test charter templateQa exploratory test charter template
Qa exploratory test charter template
 
Test Automation in Agile
Test Automation in AgileTest Automation in Agile
Test Automation in Agile
 

Similar to Quality Built In @ Spotify

Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
GoIT
 
Testing practicies not only in scala
Testing practicies not only in scalaTesting practicies not only in scala
Testing practicies not only in scala
Paweł Panasewicz
 
Agile & DevOps - It's all about project success
Agile & DevOps - It's all about project successAgile & DevOps - It's all about project success
Agile & DevOps - It's all about project success
Adam Stephensen
 
Continous integration and delivery for single page applications
Continous integration and delivery for single page applicationsContinous integration and delivery for single page applications
Continous integration and delivery for single page applications
Sunil Dalal
 
Leveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver ExampleLeveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver Example
TechWell
 
Part 2 improving your software development v1.0
Part 2   improving your software development v1.0Part 2   improving your software development v1.0
Part 2 improving your software development v1.0
Jasmine Conseil
 
Agile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery WorkshopAgile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery Workshop
Michael Palotas
 
Large scale agile development practices
Large scale agile development practicesLarge scale agile development practices
Large scale agile development practices
Skills Matter
 
Agile a to z chapter 4 feedback loop
Agile a to z chapter 4 feedback loopAgile a to z chapter 4 feedback loop
Agile a to z chapter 4 feedback loop
Wajih Aslam
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation
Sauce Labs
 
DevOps and Build Automation
DevOps and Build AutomationDevOps and Build Automation
DevOps and Build Automation
Heiswayi Nrird
 
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOps
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOpsAgile A to Z Chapter 4 Feedback Loop Part 2 DevOps
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOps
Agile ME
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
Araf Karsh Hamid
 
CookpadTechConf2018-(Mobile)TestAutomation
CookpadTechConf2018-(Mobile)TestAutomationCookpadTechConf2018-(Mobile)TestAutomation
CookpadTechConf2018-(Mobile)TestAutomation
Kazuaki Matsuo
 
Continuous integration and delivery for java based web applications
Continuous integration and delivery for java based web applicationsContinuous integration and delivery for java based web applications
Continuous integration and delivery for java based web applications
Sunil Dalal
 
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
Making software development processes to work for you
Making software development processes to work for youMaking software development processes to work for you
Making software development processes to work for you
Ambientia
 
[ENGLISH] TDC 2015 - PHP Trail - Tests and PHP Continuous Integration Enviro...
[ENGLISH] TDC 2015 - PHP  Trail - Tests and PHP Continuous Integration Enviro...[ENGLISH] TDC 2015 - PHP  Trail - Tests and PHP Continuous Integration Enviro...
[ENGLISH] TDC 2015 - PHP Trail - Tests and PHP Continuous Integration Enviro...
Bruno Tanoue
 
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
Gherkin /BDD intro
Gherkin /BDD introGherkin /BDD intro

Similar to Quality Built In @ Spotify (20)

Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
Встреча Quality built in @Spotify от Андрея Дзыни : как стать лучшим тестиров...
 
Testing practicies not only in scala
Testing practicies not only in scalaTesting practicies not only in scala
Testing practicies not only in scala
 
Agile & DevOps - It's all about project success
Agile & DevOps - It's all about project successAgile & DevOps - It's all about project success
Agile & DevOps - It's all about project success
 
Continous integration and delivery for single page applications
Continous integration and delivery for single page applicationsContinous integration and delivery for single page applications
Continous integration and delivery for single page applications
 
Leveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver ExampleLeveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver Example
 
Part 2 improving your software development v1.0
Part 2   improving your software development v1.0Part 2   improving your software development v1.0
Part 2 improving your software development v1.0
 
Agile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery WorkshopAgile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery Workshop
 
Large scale agile development practices
Large scale agile development practicesLarge scale agile development practices
Large scale agile development practices
 
Agile a to z chapter 4 feedback loop
Agile a to z chapter 4 feedback loopAgile a to z chapter 4 feedback loop
Agile a to z chapter 4 feedback loop
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation
 
DevOps and Build Automation
DevOps and Build AutomationDevOps and Build Automation
DevOps and Build Automation
 
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOps
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOpsAgile A to Z Chapter 4 Feedback Loop Part 2 DevOps
Agile A to Z Chapter 4 Feedback Loop Part 2 DevOps
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
 
CookpadTechConf2018-(Mobile)TestAutomation
CookpadTechConf2018-(Mobile)TestAutomationCookpadTechConf2018-(Mobile)TestAutomation
CookpadTechConf2018-(Mobile)TestAutomation
 
Continuous integration and delivery for java based web applications
Continuous integration and delivery for java based web applicationsContinuous integration and delivery for java based web applications
Continuous integration and delivery for java based web applications
 
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STARWEST 2010 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
 
Making software development processes to work for you
Making software development processes to work for youMaking software development processes to work for you
Making software development processes to work for you
 
[ENGLISH] TDC 2015 - PHP Trail - Tests and PHP Continuous Integration Enviro...
[ENGLISH] TDC 2015 - PHP  Trail - Tests and PHP Continuous Integration Enviro...[ENGLISH] TDC 2015 - PHP  Trail - Tests and PHP Continuous Integration Enviro...
[ENGLISH] TDC 2015 - PHP Trail - Tests and PHP Continuous Integration Enviro...
 
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
 
Gherkin /BDD intro
Gherkin /BDD introGherkin /BDD intro
Gherkin /BDD intro
 

More from Andrii Dzynia

Continuous Delivery as you want it
Continuous Delivery as you want itContinuous Delivery as you want it
Continuous Delivery as you want it
Andrii Dzynia
 
Test coaching your agile team
Test coaching your agile teamTest coaching your agile team
Test coaching your agile team
Andrii Dzynia
 
Testing at Spotify
Testing at SpotifyTesting at Spotify
Testing at Spotify
Andrii Dzynia
 
What does it mean to be a test engineer?
What does it mean to be a test engineer?What does it mean to be a test engineer?
What does it mean to be a test engineer?
Andrii Dzynia
 
Exploring your unit tests
Exploring your unit testsExploring your unit tests
Exploring your unit tests
Andrii Dzynia
 
Hermetic environment for your functional tests
Hermetic environment for your functional testsHermetic environment for your functional tests
Hermetic environment for your functional tests
Andrii Dzynia
 
Applying testing mindset to software development
Applying testing mindset to software developmentApplying testing mindset to software development
Applying testing mindset to software development
Andrii Dzynia
 
Appium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriverAppium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriver
Andrii Dzynia
 
How to Manage Testing in Dynamic World
How to Manage Testing in Dynamic WorldHow to Manage Testing in Dynamic World
How to Manage Testing in Dynamic World
Andrii Dzynia
 
ExtJS WebDriver
ExtJS WebDriverExtJS WebDriver
ExtJS WebDriver
Andrii Dzynia
 
10 правил agile тестировщика IT-Brunch
10 правил agile тестировщика IT-Brunch10 правил agile тестировщика IT-Brunch
10 правил agile тестировщика IT-Brunch
Andrii Dzynia
 
Working Software Over Comprehensive Documentation
Working Software Over Comprehensive DocumentationWorking Software Over Comprehensive Documentation
Working Software Over Comprehensive Documentation
Andrii Dzynia
 
«Самоорганизуй» себя, пока не «самоорганизовали» тебя
«Самоорганизуй» себя, пока не «самоорганизовали» тебя«Самоорганизуй» себя, пока не «самоорганизовали» тебя
«Самоорганизуй» себя, пока не «самоорганизовали» тебя
Andrii Dzynia
 
Как не нужно писать Gherkin сценарии
Как не нужно писать Gherkin сценарииКак не нужно писать Gherkin сценарии
Как не нужно писать Gherkin сценарии
Andrii Dzynia
 
iOS and Android Mobile Test Automation
iOS and Android Mobile Test AutomationiOS and Android Mobile Test Automation
iOS and Android Mobile Test Automation
Andrii Dzynia
 
10 правил Agile тестировщика
10 правил Agile тестировщика10 правил Agile тестировщика
10 правил Agile тестировщика
Andrii Dzynia
 
Что такое Exploratory Testing?
Что такое Exploratory Testing?Что такое Exploratory Testing?
Что такое Exploratory Testing?
Andrii Dzynia
 
Software testing 2.0
Software testing 2.0Software testing 2.0
Software testing 2.0
Andrii Dzynia
 
Тестировщики Vs Программисты
Тестировщики Vs ПрограммистыТестировщики Vs Программисты
Тестировщики Vs Программисты
Andrii Dzynia
 
Testing Dojo Kyiv - Testing Heuristics
Testing Dojo Kyiv - Testing HeuristicsTesting Dojo Kyiv - Testing Heuristics
Testing Dojo Kyiv - Testing Heuristics
Andrii Dzynia
 

More from Andrii Dzynia (20)

Continuous Delivery as you want it
Continuous Delivery as you want itContinuous Delivery as you want it
Continuous Delivery as you want it
 
Test coaching your agile team
Test coaching your agile teamTest coaching your agile team
Test coaching your agile team
 
Testing at Spotify
Testing at SpotifyTesting at Spotify
Testing at Spotify
 
What does it mean to be a test engineer?
What does it mean to be a test engineer?What does it mean to be a test engineer?
What does it mean to be a test engineer?
 
Exploring your unit tests
Exploring your unit testsExploring your unit tests
Exploring your unit tests
 
Hermetic environment for your functional tests
Hermetic environment for your functional testsHermetic environment for your functional tests
Hermetic environment for your functional tests
 
Applying testing mindset to software development
Applying testing mindset to software developmentApplying testing mindset to software development
Applying testing mindset to software development
 
Appium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriverAppium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriver
 
How to Manage Testing in Dynamic World
How to Manage Testing in Dynamic WorldHow to Manage Testing in Dynamic World
How to Manage Testing in Dynamic World
 
ExtJS WebDriver
ExtJS WebDriverExtJS WebDriver
ExtJS WebDriver
 
10 правил agile тестировщика IT-Brunch
10 правил agile тестировщика IT-Brunch10 правил agile тестировщика IT-Brunch
10 правил agile тестировщика IT-Brunch
 
Working Software Over Comprehensive Documentation
Working Software Over Comprehensive DocumentationWorking Software Over Comprehensive Documentation
Working Software Over Comprehensive Documentation
 
«Самоорганизуй» себя, пока не «самоорганизовали» тебя
«Самоорганизуй» себя, пока не «самоорганизовали» тебя«Самоорганизуй» себя, пока не «самоорганизовали» тебя
«Самоорганизуй» себя, пока не «самоорганизовали» тебя
 
Как не нужно писать Gherkin сценарии
Как не нужно писать Gherkin сценарииКак не нужно писать Gherkin сценарии
Как не нужно писать Gherkin сценарии
 
iOS and Android Mobile Test Automation
iOS and Android Mobile Test AutomationiOS and Android Mobile Test Automation
iOS and Android Mobile Test Automation
 
10 правил Agile тестировщика
10 правил Agile тестировщика10 правил Agile тестировщика
10 правил Agile тестировщика
 
Что такое Exploratory Testing?
Что такое Exploratory Testing?Что такое Exploratory Testing?
Что такое Exploratory Testing?
 
Software testing 2.0
Software testing 2.0Software testing 2.0
Software testing 2.0
 
Тестировщики Vs Программисты
Тестировщики Vs ПрограммистыТестировщики Vs Программисты
Тестировщики Vs Программисты
 
Testing Dojo Kyiv - Testing Heuristics
Testing Dojo Kyiv - Testing HeuristicsTesting Dojo Kyiv - Testing Heuristics
Testing Dojo Kyiv - Testing Heuristics
 

Recently uploaded

SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
Jim Mimlitz, P.E.
 
Online airline reservation system project report.pdf
Online airline reservation system project report.pdfOnline airline reservation system project report.pdf
Online airline reservation system project report.pdf
Kamal Acharya
 
Adv. Digital Signal Processing LAB MANUAL.pdf
Adv. Digital Signal Processing LAB MANUAL.pdfAdv. Digital Signal Processing LAB MANUAL.pdf
Adv. Digital Signal Processing LAB MANUAL.pdf
T.D. Shashikala
 
Metrology Book, Bachelors in Mechanical Engineering
Metrology Book, Bachelors in Mechanical EngineeringMetrology Book, Bachelors in Mechanical Engineering
Metrology Book, Bachelors in Mechanical Engineering
leakingvideo
 
IS Code SP 23: Handbook on concrete mixes
IS Code SP 23: Handbook  on concrete mixesIS Code SP 23: Handbook  on concrete mixes
IS Code SP 23: Handbook on concrete mixes
Mani Krishna Sarkar
 
Thermodynamics Digital Material basics subject
Thermodynamics Digital Material basics subjectThermodynamics Digital Material basics subject
Thermodynamics Digital Material basics subject
JigneshChhatbar1
 
Evento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recapEvento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recap
Rafael Santos
 
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
amzhoxvzidbke
 
Presentation python programming vtu 6th sem
Presentation python programming vtu 6th semPresentation python programming vtu 6th sem
Presentation python programming vtu 6th sem
ssuser8f6b1d1
 
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
sanabts249
 
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
IJAEMSJORNAL
 
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptxGJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
rogginethomas
 
Jet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdfJet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdf
KIET Group of Institutions
 
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
PradeepKumarSK3
 
Stiffness Method for structure analysis - Truss
Stiffness Method  for structure analysis - TrussStiffness Method  for structure analysis - Truss
Stiffness Method for structure analysis - Truss
adninhaerul
 
Tutorial on MySQl and its basic concepts
Tutorial on MySQl and its basic conceptsTutorial on MySQl and its basic concepts
Tutorial on MySQl and its basic concepts
anishacotta2
 
The world of Technology Management MEM 814.pptx
The world of Technology Management MEM 814.pptxThe world of Technology Management MEM 814.pptx
The world of Technology Management MEM 814.pptx
engrasjadshahzad
 
Ludo system project report management .pdf
Ludo  system project report management .pdfLudo  system project report management .pdf
Ludo system project report management .pdf
Kamal Acharya
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
YanKing2
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
yadavsuyash008
 

Recently uploaded (20)

SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
 
Online airline reservation system project report.pdf
Online airline reservation system project report.pdfOnline airline reservation system project report.pdf
Online airline reservation system project report.pdf
 
Adv. Digital Signal Processing LAB MANUAL.pdf
Adv. Digital Signal Processing LAB MANUAL.pdfAdv. Digital Signal Processing LAB MANUAL.pdf
Adv. Digital Signal Processing LAB MANUAL.pdf
 
Metrology Book, Bachelors in Mechanical Engineering
Metrology Book, Bachelors in Mechanical EngineeringMetrology Book, Bachelors in Mechanical Engineering
Metrology Book, Bachelors in Mechanical Engineering
 
IS Code SP 23: Handbook on concrete mixes
IS Code SP 23: Handbook  on concrete mixesIS Code SP 23: Handbook  on concrete mixes
IS Code SP 23: Handbook on concrete mixes
 
Thermodynamics Digital Material basics subject
Thermodynamics Digital Material basics subjectThermodynamics Digital Material basics subject
Thermodynamics Digital Material basics subject
 
Evento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recapEvento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recap
 
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
杨洋李一桐做爱视频流出【网芷:ht28.co】国产国产午夜精华>>>[网趾:ht28.co】]<<<
 
Presentation python programming vtu 6th sem
Presentation python programming vtu 6th semPresentation python programming vtu 6th sem
Presentation python programming vtu 6th sem
 
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
21CV61- Module 3 (CONSTRUCTION MANAGEMENT AND ENTREPRENEURSHIP.pptx
 
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
Best Practices of Clothing Businesses in Talavera, Nueva Ecija, A Foundation ...
 
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptxGJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
GJ WASTE TO ENERGY KOCHI PROJECT v6.pptx
 
Jet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdfJet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdf
 
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
 
Stiffness Method for structure analysis - Truss
Stiffness Method  for structure analysis - TrussStiffness Method  for structure analysis - Truss
Stiffness Method for structure analysis - Truss
 
Tutorial on MySQl and its basic concepts
Tutorial on MySQl and its basic conceptsTutorial on MySQl and its basic concepts
Tutorial on MySQl and its basic concepts
 
The world of Technology Management MEM 814.pptx
The world of Technology Management MEM 814.pptxThe world of Technology Management MEM 814.pptx
The world of Technology Management MEM 814.pptx
 
Ludo system project report management .pdf
Ludo  system project report management .pdfLudo  system project report management .pdf
Ludo system project report management .pdf
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
 

Quality Built In @ Spotify

  • 1. December 6, 2014 Andrii Dzynia Road Manager to Continuous Delivery @adzynia Quality Built In
  • 2. 2 Spotify brings you the right music for every moment! Started in 2006 (in Sweden) Now 1500+ employees, 500+ engineers Over 30 million songs available Over 20,000 songs added every day 5 development centres across the globe
  • 4. 4 Quality is a “state” …. when expectations matches the reality. Not just software defects, but stakeholders expectations as well
  • 5. Why Quality is Important? 5
  • 7. What is a way to build high quality product? 7
  • 8. 8 Do not release until it’s perfect
  • 9. 9 No one is perfect.
  • 10. Example Implementing internal back office system
  • 11. 11 Step 1 - Prototyping phase Technical spike investigation Hacking different Back-End solutions Ad-hoc design discussions Stub implementation of Front End Getting early user’s feedback on Front End
  • 12. 12 WebApp with stubs Sketches on whiteboard Whatever works to get end-user understanding
  • 13. 13 Step 2 - Setting delivery pipeline Add more unit tests Add more integration tests Add more functional tests Automate deployment configuration Add dashboards and system monitoring Clarifying End User Needs
  • 15. 15 Step 3 - Connecting the dots Connect back-end and front-end parts into End-2-End system Get user’s feedback and iterate
  • 16. 16 Keep releasing small, usable, incremental pieces
  • 17. 17 What is under the hood ?
  • 18. 18
  • 19. Organic Structure 19 @joakimsunden Scaled Agile @ Spotify
  • 21. 21 How to avoid Ice-Cream effect?
  • 22. 22 Be strict with this rule.
  • 23. Testable micro-services architecture 23 Unit Tests Component Tests Functional Tests Integration Tests System/End-2-End Tests Unit Tests Component Tests Functional Tests JS Unit Tests JS Component Tests UI Functional Tests
  • 24. 24 Unit Tests Check logic of minimal code snippet with no or mocked dependencies https://github.com/mockito/mockito https://github.com/junit-team/junit
  • 25. 25 Component Tests Very much like unit tests but emulating external components e.g. cassandra database https://github.com/jsevellec/cassandra-unit
  • 26. 26 Fake back-end for client apps Client App Fake Back-End Server Spotify Back End https://github.com/azagniotov/stubby4j https://github.com/dreamhead/moco http://jsonstub.com
  • 27. 27 Integration Tests Check an integration between components You do not need whole system to check the integration https://github.com/spotify/helios/blob/master/docs/testing_framework.md
  • 28. 28 Create Test Environment ‘on the fly’ @docker containers https://www.docker.com
  • 29. 29 Functional Tests Check services API behaviour or End user use cases http://hc.apache.org http://seleniumhq.org
  • 30. 30 Model Based Testing http://graphwalker.org
  • 32. 32 Code Review & Human Testing on Pull Request
  • 34. 34 Was not covered during this talk … Load Testing Gradual Rollouts A/B Testing Feature Toggles Monitoring/Alerting Information Radiators … but stay tuned
  • 35. 35 Test Engineering Roles and Responsibilities Technical Test Engineer Test Engineer ! = Test Automator ! = Manual Tester ~ Software Engineer in Test ~ Context Driven Tester
  • 36. 36 Technical Test Engineer Work as a software developer Advocate testability of the product Argue on software design with software engineers Help with building new/injecting existed development tools
  • 37. 37 Development Productivity Tools e.g. cli control over amazon cloud http://dashing.io Docker orchestration https://github.com/spotify/helios Cassandra Unit https://github.com/jsevellec/cassandra-unit Spoticloud Jenkins job-dsl-plugin https://github.com/mikaellanger/job-dsl-plugin Dashboards
  • 38. 38 Test Engineer Knows business domain Focused on exploring the product Free to use any programming language to test specific use case Free to use any programming language to automate his work
  • 39. 39 Mind Map as a Tool https://www.mindmup.com Product tree Scenarios Playbooks Checklists Session notes
  • 40. Quality Engineers 40 @jamesmarcusbach calls us Test Jumpers TEST JUMPERS: ONE VISION OF AGILE TESTING http://www.satisfice.com/blog/archives/1372 “RESPONSIBLE TESTER” http://www.satisfice.com/blog/archives/1364 OMEGA TESTER http://www.satisfice.com/articles/omega_tester.pdf
  • 41. 41 Any other ways to improve quality?
  • 42. 42 Break Uncertainty test ideas during healthy discussions test requirements via end users collaboration discuss system design and conner cases earlier in the planing/design meetings define definition of done
  • 43. 43 Use Google Docs collaboration for finding group consensus But have responsible person to figure out what consensus means in each case
  • 44. Social Programming 44 write “checks” during implementation shape your thoughts via pair discussions peer review before merging to master
  • 45. 45 Not mentioned Engineering Practices User Stories Planning meeting Standup meeting Open workspace TDD Refactoring etc…… http://www.extremeprogramming.org/rules.html
  • 46. Employee Early Builds Testing 46
  • 47. 47 Hackathons One week to do whatever you want Do it alone or with a team Crazy ideas comes out
  • 48. 48 QA engineer at Spotify http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/
  • 49. How to check you are on a right path ? 49
  • 50. That is easy to spot 50
  • 51. 51 Ready for a journey? Make Quality explicit Find Quality promoters Define your way of Quality improvements Set right Quality constraints Share results as early as possible Keep looking further quality improvements Probably you will never end :)
  • 52. 52 @adzynia adzynia@spotify.com Spotify Engineering Culture part 1 part 2
  • 53. and … Stop saying QA when you mean Testing Want to join the band? Check out http://www.spotify.com/jobs or @Spotifyjobs for more information