A high level introduction why and how to test within in a continuous delivery or continuous integration process

1. CD test strategies
1
26 October 2017, Hylke Stapersma, @hylke1982

2. 2
AGENDA
CD test strategies
Why continuous delivery
Why (automatic) testing
CI/CD Testing strategy
Functional testing strategies
Non functional testing strategies

3. Hylke Stapersma
Software
craftsman, Java,
CI/CD, DevOps.
Streaming data
3
Personal introduction

4. 4
Why continuous delivery

5. Code is value
Every line of code should be focused on delivering
value as soon as possible. This means the code
should be delivered to the end-user within the smallest
time window possible.
5
Why continuous delivery

6. Small increments
Take small steps that can be released more often,
instead of big steps that have a high risk of failing.
6
Why continuous delivery

7. Higher quality & quality build in
Automation makes process more reliable, instead of
having a human making mistakes automation needs to
be setup once right instead.
7
Why continuous delivery

8. Faster time to market
Software only makes money in production, this can
only be done with a strategy that delivers software
automatically as fast as possible. Reliable delivery is
key here.
8
Why continuous delivery

9. Faster feedback
Almost know instantly if the software has defects and
get notified on these defects.
9
Why continuous delivery

10. Chain of trust
Every step/test will add (automatically) add trust to a
version of a application.
10
Why continuous delivery

11. 11
Why (automatic) testing

12. End-user should not experience defects
End-users who experience defects in a production
environment will loose confidence in the product. Trust
is key in making or saving money with software.
12
Why (automatic) testing

13. Find defects early
How sooner a defect is found how cheaper it will be to
fix it. A defect in production can have severe cost and
cause reputation damages.
13
Why (automatic) testing

14. Documentation
Tests descriptions and results are great ways to
document how the system supposed to work.
Automation can give the ability to create this
completely automatically.
14
Why (automatic) testing

15. Ready for refactoring
Having a automated test setup gives confidence to
refactor and find regression problems when changing
the software.
15
Why (automatic) testing

16. Performance
Being able to say something about how the system is
going to performance under certain loads instead of
finding it out by end-users.
16
Why (automatic) testing

17. Security
Find security problems in early stage. Security
problems can lead up to penalties and reputation
damages.
17
Why (automatic) testing

18. 18
CI/CD testing strategy

19. Automate everything
Automation is a great way to eliminate unnecessary
manual repetitions. Run all tests with a minimal set of
commands.
19
CI/CD testing strategy

20. Code and version everything
Writing tests as code offers a great way to collaborate
as a team, because code can be versioned, reviewed
and changed by multiple versions.
20
CI/CD testing strategy

21. Adopt TDD strategy
Write tests before writing code. This helps to get a
better understanding of the problem, and code is
covered by test before being written.
21
CI/CD testing strategy

22. Align
Choose a test framework that aligns with the solution
being build. For example choose the same
programming language for application and tests.
22
CI/CD testing strategy

23. Automate documentation
Every step that runs test should create documentation
automatically, use, share and distribute this
documentation in a automated fashion.
23
CI/CD testing strategy

24. Fail fast – fail early
Validate if every condition is met before starting
‘expensive’ tests. For example see if application is
available or there is enough disk space.
24
CI/CD testing strategy

25. 25
Functional testing
strategy

26. Test pyramid
Use the testing pyramid for a fast but complete test
setup. Try to cover almost everything with ‘cheap’ unit
tests and only do ’expensive’ the necessary
functional/acceptance tests.
26
Functional testing strategy

27. Unit testing
One unit tests should focus a single atomic part of the
application in a certain state. (good practice is one
assert per test). Execute thousands per second and
use a TDD approach.
27
Functional testing strategy

28. Mutation testing
To validate the validity of unit test use mutation testing.
See if tests fail when a line of code is changed. Will
prove the strength of the unit tests.
28
Functional testing strategy

29. Property based testing
Instead of example use properties to validate. For
example everything in a range is true. This will validate
the range.
29
Functional testing strategy

30. Integration/component testing
Validate if software components work together.
30
Functional testing strategy

31. Contract testing
Validate if interfaces are compliant on the server and
on the client side. Share documentation about
assumptions of a API. Use PACT for example.
31
Functional testing strategy

32. Functional / Acceptance testing
Describe on a high level assumptions how to end-user
will interact with the system. Use a language that can
be understood by machine and humans.
32
Functional testing strategy

33. Manual testing
Only focus manual testing on things that are hard to
understand for a computer. For example does look
right and does the response feels right.
33
Functional testing strategy

34. 34
Non functional testing
strategy

35. Security testing
Based on known security issues try to break the
application.
35
Non functional testing strategy

36. Performance testing
Find the limits, normal behavior, stress behavior and
sizing for the application. Works great by adopting a
cloud strategy.
36
Non functional testing strategy

37. Compliance testing
Automatically validate if the system is compliant with
internal and external rules.
37
Non functional testing strategy

38. Chaos monkey / Resilience testing
By introducing stress or failure find weak spots within
a system.
38
Non functional testing strategy

39. 39
Thank you