Talking about how Songkick manages software testing whilst following a Continuous Deployment approach.

1. TESTING IN A CONTINUOUS
DELIVERY WORLD
Amy Phillips

2. Amy Phillips
! @itjustbroke
!
! testingthemind.wordpress.com
2

3. Songkick

4. Testing:
4
A search for information

5. Quality:
5
Value to some person
!
- Gerald Weinberg

6. 6
Test that you rolled a one

7. 7
Test that you rolled a one

8. 8
Test that you rolled a one

9. 9
Test that you rolled a one

10. 10
Test that you rolled a one

11. 11
Test that you rolled a one

12. What were we doing?
! Checking the result
!
!
!
! Testing the result
12

13. Explore It!: Reduce
Risk and Increase
Confidence with
Exploratory Testing
Elisabeth Hendrickson
13

14. ! You can automate 100% of your system
testing
!
! Automation will guarantee your system
keeps working - after all you are
testing literally everything
14
Some Myths about Automation

15. ! You can never test everything to prove
it is working
!
! It takes just one test to prove that a
system is broken
15
Does it work?

16. ! Not working according to requirements
! Doesn’t meet the user’s expectations
! Difficult to use
! Inconsistent
! Doesn’t scale
! Can’t be maintained
! Difficult to test
! …
16
What is a bug?

17. 17
The (old) Songkick Way
Automated
Test and build
Manual Test
Feature &
ReleaseQueued
for
commit
Commit Deploy

18. 18
Don’t Aim for Perfection

19. Why do we have this process?
Release
Problems
Make
everything
safe

20. Scheduled
Continuous
Releases
Integration
!
!
!
20
What were our options?
or

21. What are our values?
!
! Ship new features as soon as possible
!
! Developers responsible for production
quality

22. 22
The (old) Songkick Way
Automated
Test and build
Manual Test
Feature &
ReleaseQueued
for
commit
Commit Deploy
1 Hour 2 Hours

23. Define a single test strategy
Developers Testers
Unit Tests
Integration Tests
Manual Checks
Manual Tests
Regression Tests
Automated Checks

24. Define a single test strategy
Developers Testers
Unit Tests
Integration Tests Manual Checks
Manual Tests
Regression Tests
Automated Checks

25. 25
!
Testing
100 %Team Effort
CheckingTesting
Agree on how to split test effort

26. 26
!
100 %Team Effort
Testing Machine Human
Agree on how to split test effort

27. 27
!
!
! Dev teams are responsible for their
own testing
A New Value

28. 28
!
! Test at the right level
! Everyone understands what is being
tested and why
! Everyone has a chance to influence
what should be tested
! Fast feedback
Shared ownership of Automated
tests

29. 29
!
!
! What are we hoping to achieve with
this change?
! Identify risks
! Agree on how to mitigate risks
Risk Assess Everything

30. 30
The Songkick Way - low risk
Automated
Test and
build
Commit DeployAutomated
Regression

31. 31
The Songkick Way - medium
risk
Automated
Test and
build
Manual
Test on
Dev env
Commit DeployAutomated
Regression

32. 32
The Songkick Way - high risk
Automated
Test and
build
Manual
Test on
Dev env
Commit DeployAutomated
Regression
Manual
Test on
Prod env

33. Releasing doesn’t have to mean
Launching

34. Trust the Test Results
! Never release if the build is red
! Green builds must mean the tests have
passed

35. !
!
You can break anything once
!
- Songkick Maxim
35
Learn from your mistakes

36. Executing the
right tests
at the right time
Automated build
and
deploy process
Continuous Deployment

37. 37
Tools

38. 38
Use your brain

39. 39
Keep on talking

40. Releases
0
28
56
84
112
140
Mar 2011 Jul 2011 Nov 2011 Mar 2012 Jul 2012

41. But this ended up being about
more than just releases…
! Everyone cares about testing
! Fast and maintained automated tests
! Bug fixes often take just minutes

42. In Summary
! You must automate
! But don’t try to automate everything
! The automation you have doesn’t
necessarily make you safe
! Use problems to drive positive change
! Don’t neglect the human aspect

43. 43
Songkick
is hiring

44. 44
Questions?
! @itjustbroke
!
! testingthemind.wordpress.com
!
! http://devblog.songkick.com/