Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing in a Continuous World


Published on

Workshop given at Agile Testing Days 2017 together with Lisa Crispin: "Testing in a Continuous World - How to Continuously Learn for Successful Continuous Delivery"


Nowadays users are quite used to getting product updates every other day. Many teams want to release updates to production more frequently, but they’re struggling to complete testing activities and fear that they might deliver big bugs instead of valuable features. Continuous delivery of reliable software is a huge challenge.

Many teams already develop their products in small chunks, but how does testing and quality fit into this fast-paced scenario? How can we not only continuously deliver, but continuously provide value to our users?

Elisabeth and Lisa have worked in teams who struggled to have testing fully integrated, being able to only release once every few weeks. They’ve also experienced highly collaborative teams who could easily deliver multiple times a day to quickly validate if they added value. What's your situation, can you keep up with testing and release frequently?

In this hands-on workshop, participants will have a chance to practice techniques that can help teams feel confident releasing more frequently. We will facilitate activities in which participants will come up with new experiments to help shorten feedback cycles, make sure all essential types of testing are done continually, and fit testing into the continuous world. Let's share experiences and develop custom solutions for your specific scenarios together!

Published in: Software
  • Be the first to comment

Testing in a Continuous World

  1. 1. Welcome to Testing in a Continuous World! Before we start: • Sit with people with different jobs or experience level from yours • Start writing down your challenges around continuous delivery, one per sticky note (more time for this later) @LISACRISPIN | @LISIHOCKE
  4. 4. Setting expectations WHAT WE WON’T DO • Tell you the one and only solution • Provide a blueprint to copy • Lecture you WHAT WE WILL DO INSTEAD • Give some inspiration what you could try • Provide a tool you can use after you returned to work • Facilitate a hands-on, interactive and collaborative exercise @LISACRISPIN | @LISIHOCKE
  5. 5. 1. Identify real-life challenges for testing in a continuous delivery environment 2. Design real-life experiments to address those challenges 3. Take the experiments and try them at your workplace Learning intentions @LISACRISPIN | @LISIHOCKE
  6. 6. Continuous delivery • Release software reliably and without friction (Martin Fowler) • Production deploys should be boring (Jez Humble) • How can we feel confidence for frequent deploys? @LISACRISPIN | @LISIHOCKE
  7. 7. Real-life stories • Real teams; in the following named Team Orange & Team Blue • Real challenges • Real hypotheses • Real experiments • Real results @LISACRISPIN | @LISIHOCKE
  8. 8. Team Orange: Challenge Challenge: The team has a perception that quality can be assured by one person at the end @LISACRISPIN | @LISIHOCKE
  9. 9. Team Orange: Hypothesis I believe that renaming the “QA Review” column to “Final Testing” and breaking out of the perceived role limits will result in the team’s change of perception from assuring quality towards building quality in by the whole team, and from testing late towards testing throughout the workflow. I’ll know I have succeeded when I don’t hear about quality assurance anymore or get questioned why starting testing early after six months. @LISACRISPIN | @LISIHOCKE
  10. 10. Team Orange: Experiment • Rename column to “Final testing” to reflect only last checks on story level before merging • Start testing as early as possible, test incrementally, also on production • Role rebranding: changed title from “QA Developer” to “Agile Tester” @LISACRISPIN | @LISIHOCKE
  11. 11. Team Orange: Results • Team started talking about testing as activity instead of QA as phase or role • Team stopped questioning testing throughout the workflow • Team appreciated early feedback • Increased transparency of board @LISACRISPIN | @LISIHOCKE
  12. 12. Team Blue: Challenge Challenge: • Many regressions after move to twice-weekly deploys • Still had manual release regression checklists • Many release-stopper bugs, bad regressions getting into production @LISACRISPIN | @LISIHOCKE
  13. 13. Team Blue: Hypothesis We believe that pairing in exploratory testing and automation will result in a reduced frequency of production release rollbacks and hotfixes. We’ll know we succeeded when the average number of release rollbacks and hotfixes decreases from 5 per month to 3 per month within 3 months. @LISACRISPIN | @LISIHOCKE
  14. 14. Team Blue: Experiment • Stop doing manual release regression testing • Developer pairs do exploratory testing on each story • Testers pair at least 1 day per week with developers @LISACRISPIN | @LISIHOCKE
  15. 15. Team Blue: Results • Average went down to 3 per month for 3 months • But has crept back up to 5 • Many factors involved • Tester/developer pairing now established practice, but trying more experiments @LISACRISPIN | @LISIHOCKE
  16. 16. How to test continuously, successfully? What is key: • Visibility • Shared understanding • Short feedback loops → Let’s look at just a few practices that can help @LISACRISPIN | @LISIHOCKE
  17. 17. Visibility Some visualization techniques: • Draw and discuss together • Mind maps • System architecture • Workflow & pipelines @LISACRISPIN | @LISIHOCKE
  18. 18. Shared understanding Some framework examples: • Behavior-Driven Development • Example mapping • Specification workshops • User story mapping @LISACRISPIN | @LISIHOCKE
  19. 19. Short feedback loops FROM CLOSE COLLABORATION • Stop starting, start finishing • Cross-functional pairing • Mobbing FROM TESTS • TDD • Adequate automated test coverage • Exploratory testing @LISACRISPIN | @LISIHOCKE FROM USERS • Feature toggles • Monitoring in production • Analytics • Direct feedback FROM INFRASTRUCTURE • Continuous integration • Spin up test environments • Deploy automatically on green build
  21. 21. Exercise Part 1: Brainstorm challenges @LISACRISPIN | @LISIHOCKE
  22. 22. Exercise Part 1: Brainstorm challenges 1. Write down your challenges around continuous delivery 2. One challenge per sticky note @LISACRISPIN | @LISIHOCKE
  23. 23. Exercise Part 2: Choose a challenge @LISACRISPIN | @LISIHOCKE
  24. 24. Exercise Part 2: Choose a challenge A bit like Lean Coffee: • Go around the table, each person gets 30 seconds to explain their challenge to others in the group • Group similar topics together • After everyone has explained their topics, dot vote to choose a challenge to work on – 3 votes per person • Prioritize topics by votes • Start with the top challenge (if time is left you can work on more than one) @LISACRISPIN | @LISIHOCKE
  25. 25. Exercise Part 3: Design your experiment! @LISACRISPIN | @LISIHOCKE
  26. 26. Exercise Part 3: Design your experiment! For your chosen challenge: • Create a measurable hypothesis with a prediction • Design an experiment to test the hypothesis in the chosen work context (be specific) • Make a poster showing challenge, hypothesis & experiment Challenge: ______________________ Hypothesis: • We believe that __________________ • will result in _____________________ • We’ll know we have succeeded when _____________________________ Experiment: • _____________________________ • _____________________________ @LISACRISPIN | @LISIHOCKE
  27. 27. Exercise Part 4: World Café @LISACRISPIN | @LISIHOCKE
  28. 28. • Choose a spokesperson to take your poster and travel to the next table on the left • Everyone else stays at their table • Spokespeople explain the challenge, hypothesis and experiment to the visited group • The group gives feedback and contributes their ideas for the presented experiment Exercise Part 4: World Café @LISACRISPIN | @LISIHOCKE
  29. 29. Exercise Part 5: Incorporate feedback @LISACRISPIN | @LISIHOCKE
  30. 30. Exercise Part 5: Incorporate feedback • Tweak your hypothesis and experiment based on the feedback you got from others • Make a new poster if needed • Get ready to present your challenge, hypothesis and experiment to everyone ☺ @LISACRISPIN | @LISIHOCKE
  31. 31. Exercise Part 6: Group presentations @LISACRISPIN | @LISIHOCKE
  32. 32. Exercise Part 6: Group presentations You have 1min per group to present… • the challenge, • the measurable hypothesis, • the designed experiment, • the most valuable idea you got from the other groups that you decided to incorporate, • and who will run the experiment to test the hypothesis back at work! @LISACRISPIN | @LISIHOCKE
  34. 34. 1. Identify real-life challenges for testing in a continuous delivery environment 2. Design real-life experiments to address those challenges 3. Take the experiments and try them at your workplace Learning intentions reviewed @LISACRISPIN | @LISIHOCKE
  35. 35. Questions? @LISACRISPIN | @LISIHOCKE
  36. 36. Save 35%* off the books or ebooks eBook formats include EPUB, MOBI, & PDF See for more Save 50%* on Agile Testing Essentials LiveLessons Video Training Use code AGILETESTING at *Discount taken off list price. Offer only good at and is subject to change. @LISACRISPIN | @LISIHOCKE