Workshop given at Agile Testing Days 2017 together with Lisa Crispin: "Testing in a Continuous World - How to Continuously Learn for Successful Continuous Delivery" Abstract: Nowadays users are quite used to getting product updates every other day. Many teams want to release updates to production more frequently, but they’re struggling to complete testing activities and fear that they might deliver big bugs instead of valuable features. Continuous delivery of reliable software is a huge challenge. Many teams already develop their products in small chunks, but how does testing and quality fit into this fast-paced scenario? How can we not only continuously deliver, but continuously provide value to our users? Elisabeth and Lisa have worked in teams who struggled to have testing fully integrated, being able to only release once every few weeks. They’ve also experienced highly collaborative teams who could easily deliver multiple times a day to quickly validate if they added value. What's your situation, can you keep up with testing and release frequently? In this hands-on workshop, participants will have a chance to practice techniques that can help teams feel confident releasing more frequently. We will facilitate activities in which participants will come up with new experiments to help shorten feedback cycles, make sure all essential types of testing are done continually, and fit testing into the continuous world. Let's share experiences and develop custom solutions for your specific scenarios together!

1. Welcome to Testing in a
Continuous World!
Before we start:
• Sit with people with
different jobs or
experience level from
yours
• Start writing down your
challenges around
continuous delivery, one
per sticky note (more time
for this later)
@LISACRISPIN | @LISIHOCKE

2. Testing in a
Continuous World
HOW TO CONTINUOUSLY LEARN FOR
SUCCESSFUL CONTINUOUS DELIVERY
@LISACRISPIN | @LISIHOCKE

3. Introduction
SETTING THE STAGE
@LISACRISPIN | @LISIHOCKE

4. Setting expectations
WHAT WE WON’T DO
• Tell you the one and only
solution
• Provide a blueprint to
copy
• Lecture you
WHAT WE WILL DO INSTEAD
• Give some inspiration
what you could try
• Provide a tool you can use
after you returned to work
• Facilitate a hands-on,
interactive and
collaborative exercise
@LISACRISPIN | @LISIHOCKE

5. 1. Identify real-life challenges for testing in a continuous
delivery environment
2. Design real-life experiments to address those
challenges
3. Take the experiments and try them at your workplace
Learning intentions
@LISACRISPIN | @LISIHOCKE

6. Continuous delivery
• Release software reliably and without friction (Martin
Fowler)
• Production deploys should be boring (Jez Humble)
• How can we feel confidence for frequent deploys?
@LISACRISPIN | @LISIHOCKE

7. Real-life stories
• Real teams; in the
following named
Team Orange & Team Blue
• Real challenges
• Real hypotheses
• Real experiments
• Real results
@LISACRISPIN | @LISIHOCKE

8. Team Orange: Challenge
Challenge:
The team has a perception that quality can be assured by
one person at the end
@LISACRISPIN | @LISIHOCKE

9. Team Orange: Hypothesis
I believe that renaming the “QA Review” column to “Final
Testing” and breaking out of the perceived role limits
will result in the team’s change of perception from assuring
quality towards building quality in by the whole team, and
from testing late towards testing throughout the workflow.
I’ll know I have succeeded when I don’t hear about quality
assurance anymore or get questioned why starting testing
early after six months.
@LISACRISPIN | @LISIHOCKE

10. Team Orange: Experiment
• Rename column to “Final
testing” to reflect only last
checks on story level
before merging
• Start testing as early as
possible, test
incrementally, also on
production
• Role rebranding: changed
title from “QA Developer”
to “Agile Tester”
@LISACRISPIN | @LISIHOCKE

11. Team Orange: Results
• Team started talking about testing as activity instead of
QA as phase or role
• Team stopped questioning testing throughout the
workflow
• Team appreciated early feedback
• Increased transparency of board
@LISACRISPIN | @LISIHOCKE

12. Team Blue: Challenge
Challenge:
• Many regressions after move to twice-weekly deploys
• Still had manual release regression checklists
• Many release-stopper bugs, bad regressions getting into
production
@LISACRISPIN | @LISIHOCKE

13. Team Blue: Hypothesis
We believe that pairing in exploratory testing and
automation
will result in a reduced frequency of production release
rollbacks and hotfixes.
We’ll know we succeeded when the average number of
release rollbacks and hotfixes decreases from 5 per month
to 3 per month within 3 months.
@LISACRISPIN | @LISIHOCKE

14. Team Blue: Experiment
• Stop doing manual release regression testing
• Developer pairs do exploratory testing on each story
• Testers pair at least 1 day per week with developers
@LISACRISPIN | @LISIHOCKE

15. Team Blue: Results
• Average went down to 3 per month for 3 months
• But has crept back up to 5
• Many factors involved
• Tester/developer pairing now established practice, but
trying more experiments
@LISACRISPIN | @LISIHOCKE

16. How to test continuously,
successfully?
What is key:
• Visibility
• Shared understanding
• Short feedback loops
→ Let’s look at just a few
practices that can help
@LISACRISPIN | @LISIHOCKE

17. Visibility
Some visualization techniques:
• Draw and discuss together
• Mind maps
• System architecture
• Workflow & pipelines
@LISACRISPIN | @LISIHOCKE

18. Shared understanding
Some framework examples:
• Behavior-Driven Development
• Example mapping
• Specification workshops
• User story mapping
@LISACRISPIN | @LISIHOCKE

19. Short feedback loops
FROM CLOSE COLLABORATION
• Stop starting, start
finishing
• Cross-functional pairing
• Mobbing
FROM TESTS
• TDD
• Adequate automated test
coverage
• Exploratory testing
@LISACRISPIN | @LISIHOCKE
FROM USERS
• Feature toggles
• Monitoring in production
• Analytics
• Direct feedback
FROM INFRASTRUCTURE
• Continuous integration
• Spin up test environments
• Deploy automatically on
green build

20. Exercise
GROUP WORK IN SIX ACTS
@LISACRISPIN | @LISIHOCKE

21. Exercise Part 1: Brainstorm challenges
@LISACRISPIN | @LISIHOCKE

22. Exercise Part 1: Brainstorm challenges
1. Write down your challenges around continuous delivery
2. One challenge per sticky note
@LISACRISPIN | @LISIHOCKE

23. Exercise Part 2: Choose a challenge
@LISACRISPIN | @LISIHOCKE

24. Exercise Part 2: Choose a challenge
A bit like Lean Coffee:
• Go around the table, each person gets 30 seconds to
explain their challenge to others in the group
• Group similar topics together
• After everyone has explained their topics, dot vote to
choose a challenge to work on – 3 votes per person
• Prioritize topics by votes
• Start with the top challenge (if time is left you can work
on more than one)
@LISACRISPIN | @LISIHOCKE

25. Exercise Part 3: Design your experiment!
@LISACRISPIN | @LISIHOCKE

26. Exercise Part 3: Design your experiment!
For your chosen challenge:
• Create a measurable
hypothesis with a
prediction
• Design an experiment to
test the hypothesis in the
chosen work context (be
specific)
• Make a poster showing
challenge, hypothesis &
experiment
Challenge: ______________________
Hypothesis:
• We believe that
__________________
• will result in
_____________________
• We’ll know we have succeeded
when
_____________________________
Experiment:
• _____________________________
• _____________________________
@LISACRISPIN | @LISIHOCKE

27. Exercise Part 4: World Café
@LISACRISPIN | @LISIHOCKE

28. • Choose a spokesperson to take your poster and travel to
the next table on the left
• Everyone else stays at their table
• Spokespeople explain the challenge, hypothesis and
experiment to the visited group
• The group gives feedback and contributes their ideas for
the presented experiment
Exercise Part 4: World Café
@LISACRISPIN | @LISIHOCKE

29. Exercise Part 5: Incorporate feedback
@LISACRISPIN | @LISIHOCKE

30. Exercise Part 5: Incorporate feedback
• Tweak your hypothesis and experiment based on the
feedback you got from others
• Make a new poster if needed
• Get ready to present your challenge, hypothesis and
experiment to everyone ☺
@LISACRISPIN | @LISIHOCKE

31. Exercise Part 6: Group presentations
@LISACRISPIN | @LISIHOCKE

32. Exercise Part 6: Group presentations
You have 1min per group to present…
• the challenge,
• the measurable hypothesis,
• the designed experiment,
• the most valuable idea you got from the other groups that
you decided to incorporate,
• and who will run the experiment to test the hypothesis
back at work!
@LISACRISPIN | @LISIHOCKE

33. Closing
WRAP-UP. QUESTIONS? GIVEAWAYS! EVALUATIONS
@LISACRISPIN | @LISIHOCKE

34. 1. Identify real-life challenges for testing in a continuous
delivery environment
2. Design real-life experiments to address those
challenges
3. Take the experiments and try them at your workplace
Learning intentions reviewed
@LISACRISPIN | @LISIHOCKE

35. Questions?
@LISACRISPIN | @LISIHOCKE

36. Save 35%* off the books or ebooks
eBook formats include EPUB, MOBI, & PDF
See www.agiletester.ca for more
Save 50%* on Agile Testing Essentials
LiveLessons Video Training
Use code AGILETESTING at
informit.com/agile
*Discount taken off list price. Offer only good at
informit.com and is subject to change.
@LISACRISPIN | @LISIHOCKE