Automatic tests are fundamental in a continuous delivery pipeline, but exactly what do you want to achieve? How should you prioritize? How will you handle all the test results you get? Without a test strategy, you will soon be lost, but this slide show gives you some hints on the way and some questions to which you will have to find your own answers. Listen to the presentation on YouTube: Praqma Channel.

1. Test Automation
Do we still need test specialists?
Håkan Rönngren, T2 Data AB

2. Who am I?
2000 2010
Continuous Delivery
Test Automation
Software Development

3. The next 28 minutes...
Why do we want automatic tests?
Some things to consider

4. Goals of the Game

5. Immediate Feedback
Dev Test
Dev Test Dev TestDev Test Dev Test Dev Test

6. Immediate Feedback
The faster and cheaper the test, the closer to
the developer it should run.
❖ Development Environment
❖ Hooks in Revision Control
❖ Review Tools
❖ Scheduled Builds

7. Become able to test more
Hardware Variants
Software Versions
Test Suites

8. Where does the ROI come from?
Early bug discovery
Accuracy
Visibility
Coverage
New tests (load, stress…)
More interesting work

9. Challenges

10. Don’t just automate manual tests
Briefly written, require knowledge
Incomplete, require investigation
Not modular, no reuse

11. Don’t expect to automate everything
Manual end-to-end
Acceptance
Exploratory testing
Just difficult

12. Test Strategy
Where does your automatic testing fit in?
What does it require?
Test objectives
Test levels
Test groups
Priorities
Roles & Responsibilities
Change Management
Test Environments
Test Tools
Risks and Mitigations
Scheduling
Traceability
Test Reporting

13. Don’t get dazzled by tools
❖Have specific expectations
❖Prioritize according to your test strategy

14. The Infamous Metrics Trap
Number of Test Cases ≠ Test Effectiveness
Coverage Measure ≠ Test Effectiveness

15. Tests are code
Gestures may be hard to maintain
Programming skills needed
Revision control

16. Focus on maintainability
In an agile world, things change all the time
Make sure your tests are:
❖Clearly structured, not redundant
❖Comprehensible and clearly scoped

17. Improve your testability
Preparable Controllable Observable

18. Training
Include everybody
❖What is being tested?
❖How do I get and interpret the results?
❖What am I supposed to contribute?
❖How can I solve problems?

19. Look at the whole picture
Higher test tempo means much more data
❖What features are in this test run?
❖What test results did my commit get?
❖Who committed to this test run?
❖What test results did this delivery get?
❖Is this failure new/known/recurring?
❖Is this bug report fix verified by a test?

20. Improve continuously
If you are not going up, you are going down.
Take a LEAN perspective on your automatic
testing.

21. Conclusions
❖Know what you want
❖Also know how you expect to get it
❖Take a holistic approach
❖A complement, not a substitute
❖Requires time, investments and training
❖Focus on maintainability
❖Never stop improving

22. Any money to save here?
“Okay, so here is the deal.
You write that test
automation thingy that does
your job, and then I fire you.”
(Yes, probably a lot. Fire
that boss immediately.)