Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dare to Explore: Discover ET!


Published on

Gain a deeper understanding of what Exploratory Testing (ET) is, the essential elements of the practice with practical tips and techniques, and finally, ideas for integrating ET into the cadence of an agile process

Published in: Software
  • Be the first to comment

Dare to Explore: Discover ET!

  1. 1. Dare to Explore… Discover ET!
  2. 2. Outline •  What Exploratory Testing (ET) is and isn’t •  Dissecting Exploratory Testing – Learning – Designing – Executing •  Fitting it within the Agile Context •  Adoption Experience •  Q & A
  3. 3. What does Exploratory Testing mean to YOU?
  4. 4. Testing Pyramid Integration Tests Unit Tests Developer-centric Are we building the code right? Business-centric Are we building the right code? Adaptation of Mike Cohn's test automation pyramid Exploratory Testing GUI Tests Acceptance Tests (API layer) verification
  5. 5. What is Exploratory Testing? Test-related learning Test design Test execution Test result interpretation
  6. 6. Exploratory Testing is a mindset Embrace continuous improvement of test quality Enabling choice, not constraining it Tests are experiments for learning Push the boundaries, don’t execute routine checks Embrace continuous learning
  7. 7. Exploratory Testing… IS NOT IS Doing random stuff to see what happens A rigorous investigative practice Repetitive Adaptive Replacement for predesigned testing Complimentary to conventional techniques About producing comprehensive test cases for later use Creative & more free-style
  8. 8. Why ET?
  9. 9. Two sides of Testing Tested Checked (scripted manual + automated) Explored “No matter how many tests we write, no matter how many cases we execute, we always find the most serious bugs when we go off the script” Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson
  10. 10. Checking •  Designing scripts with expected results upfront •  Verifies what was working still works •  Well documented and repeatable Exploring •  Learning, designing and executing at the same time •  Tests the system capabilities, boundaries •  Can be sparsely documented and hard to repeat
  11. 11. Exploration with a general mission without specific step- by-step instructions Effectiveness relies on the tester’s knowledge, skills, and experience Simultaneously learning about the software under test while designing and executing tests, using feedback from the last test to inform the next. Characteristics of ET
  12. 12. ET is a fit where… •  Need for manual testing approach that’s adaptable because the SUT changes rapidly •  Requirements change often, and the specifications are vague or incomplete •  Writing detailed test scripts for a manual test effort doesn’t make sense •  Need to provide rapid feedback on a new product or feature
  13. 13. Key testing problem •  We must test a product whose purpose is emerging and/or changing •  Coded without fully specified requirements upfront   Agile fit?
  14. 14. ET: Nuts & Bolts
  15. 15. The Flow… •  Focus the exploration •  Learn about the system/features •  Design tests with interesting variations •  Attack and report Unscripted doesn’t mean unprepared  
  16. 16. Test charters bring Focus Charters define the area(s) we are testing and the kind of vulnerabilities we’re looking for -  Features -  Suspected risky areas of the system -  Coupling points Meandering without direction/purpose is getting lost  
  17. 17. Sample Charter •  Target: What are you exploring? Feature, component, interface etc. •  Resources: Data set, technique, configuration etc. •  Information: What are you hoping to find? Is it security vulnerability, performance, capability, usability etc. Explore <target> With <resources> To discover <information> Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson Explore the Foo Form Fee Payment functionality using active accounts in the Demo environment to discover how the payment interface responds. Template Example
  18. 18. Learn the general shape What does the system do? Input, Store, Compute, Output “I know that I do not know” ~ Socrates  
  19. 19. Design tests with interesting variations The narrower the view, the wider the ignorance   •  Varying sequences of actions and varying timing •  Test error handling by making required resources unavailable or locked, or to break connections •  Inventing personae to generate extreme scenarios •  ...
  20. 20. Use heuristics to guide test design on the fly   •  Test Heuristics Cheat Sheet: (Elisabeth Hendrickson, James Lyndsay, and Dale Emery) testheuristicscheatsheetv1.pdf •  How To Break Software: “17 Attacks” (Alan A. Jorgensen, James A. Whittaker)    
  21. 21. Ideas for Executing & Recording All code is guilty, until proven innocent. •  Explore Early, Explore Often •  Time-box the session •  Line up the team •  Surfacing potential problems/risks quickly is the goal •  Work in pairs or groups, whenever possible •  Be distracted by anomalies and new ideas •  Avoid test repetition •  Document your findings in an appropriate way (low overhead is key)  
  22. 22. Running ET Sessions
  23. 23. Moving Quality Concerns Upstream Previous State Bug bash and IV&V executed just before the release Target State Using a whole team approach to exploring early and often, we hope to surface and address risks sooner, and that can lead to better software, fewer surprises, and greater confidence
  24. 24. Approach •  Synchronized Cadence across 8 Agile teams •  Test Charter Autonomy •  Time-boxed Exploratory Sessions -  Explore & Record Observations -  Summarize and Report
  25. 25. Current   Enterprise     Comprised  of  many  disparate   data  sources  and  systems,   which  leads  to:   Inconsistent  Data   Data  is  defined,  captured  and  interpreted  inconsistently  throughout  the  Enterprise   Ad-­‐Hoc  and  Near  Real-­‐Time  Analysis  Limita<ons   Limited  ability  to  conduct  ad-­‐hoc  and  near  real-­‐>me  analysis,  delaying  >mely     decision-­‐making   Cross-­‐Func<onal  Domain  Limita<ons   Inability  to  provide  a  cross-­‐func>onal  view  across  data  domains  easily  or  completely   Lack  of  Agility  in  Response  to     Evolving  Business  Model   Lack  of  agility  to  adjust  data  procurement  and  delivery  in  response  to  a  rapidly  evolving  business   model   Lack  of  Data  Across  the  Enterprise   Data  gaps  exist  due  to  limited  understanding  of  current  and  an>cipated  data  usage  and   applica>on  across  the  Enterprise   Notes From the Trenches Challenges and solutions transitioning a traditional test team into an exploratory test team on a large, heavily interfaced system. I’ve a feeling we’re not in Kansas anymore Challenge: Changing a mindset Solution: Framework, cadence, training, discussions, and metrics Why do you want access to my system? Challenge: You need “visas” to explore Solution: You need allies, and strong ones Yet another build!!!? Challenge: Continuous Delivery means continuous deployments Solution: Test in parallel with and after production deployments What is truth? Challenge: Acceptance criteria vs. Desired Outcome Solution: Find someone who cares Boldly going where no one has gone before Challenge: Indigenous population may be indigent Solution: “Marketing” brochure and chocolate chip cookies
  26. 26. Concluding thoughts
  27. 27. Benefits Unleashes tester’s creativity and skill Ability to perform focused testing without comprehensive documentation Rapid flow of feedback to the team Based  on  the  summary  by  Itkonen  and  Rau>ainen  
  28. 28. Challenges Managing test coverage can be difficult Quality of testing highly dependent on expertise and skill of tester Repeatability of found defects challenging at times
  29. 29. Summary •  Surface and address risks sooner •  Can help identify the big bugs •  Adaptive •  Complementary to scripted testing •  Agile fit
  30. 30. 30  
  31. 31. Raj Indugula   John Hughes  
  32. 32. Resources and References •  Bach, James. “What Is Exploratory Testing?” •  Marick, Brian. “A Survey of Exploratory Testing” •  Hendrickson, Elisabeth. “Explore It!” •  Jorgensen, Alan. “How to Break Software (with examples)” •  Kaner, Cam, Tinkham, Andy. “Learning Styles and Exploratory Testing” •  Bolton, Michael, et al. “Exploratory Testing Dynamics” •  Itkonen, Juha. “Do test cases really matter? An experiment comparing test case based and exploratory testing” 32