Michael mahlberg exploratory-testing-the_missing_half_of_bdd

2,388 views

Published on

"We should just call it testing - when it's not exploratory testing it's not real testing anyway" -Twitter, Summer 2011 Lately many professional testers have started to make a clear distinctions between thing that we call testing (like TDD and BDD) and what they consider testing - referring to TDD and BDD mostly as checking. And actually I – and I would think many of you as well – have seen projects with a test coverage of 80% and more that still fail to meet the clients' needs. Even though they meet the specifications perfectly. This points to some value that could be added to techniques like BDD and TDD by embracing the ideas from people like James Marcus Bach, Paul Carvalho and Michael Bolton. After giving an overview of current trends in the testing community like ET (exploratory testing) and ATDD (Acceptance Test Driven Development) this session will try to do exactly that: discuss the - often missing - intersection between BDD and exploratory testing and suggest ways to fill it.

Published in: Technology, Education
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
2,388
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
62
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

  1. 1. Michael Mahlberg, 05.06.2012 Exploratory Testing – The missing half of BDD?Tuesday, June 5, 2012
  2. 2. Testing?Tuesday, June 5, 2012
  3. 3. AGAIN?Tuesday, June 5, 2012
  4. 4. It’s not about Rails http://www.flickr.com/photos/chris_gin/2490980153/Tuesday, June 5, 2012
  5. 5. Several StoriesTuesday, June 5, 2012
  6. 6. BDD Specification Design by TDD by Example Contract Defensive Programming Software- DevelopmentTuesday, June 5, 2012
  7. 7. Exploratory TestingTuesday, June 5, 2012
  8. 8. Tuesday, June 5, 2012
  9. 9. Tuesday, June 5, 2012
  10. 10. Tuesday, June 5, 2012
  11. 11. Tuesday, June 5, 2012
  12. 12. Paul Carvalho @can_testTuesday, June 5, 2012
  13. 13. Michael Bolton No, Not this oneTuesday, June 5, 2012
  14. 14. Michael Bolton @michaelboltonTuesday, June 5, 2012
  15. 15. Markus Gärtner @mgaertneTuesday, June 5, 2012
  16. 16. Players in the Game • James Bach • Michael Bolten (not the singer) • Cem Kaner • Paul CarvalhoTuesday, June 5, 2012
  17. 17. Ressources Human Resource Paul Carvalho http://swtester.blogspot.de/ Michael Bolton http://www.developsense.com/blog/ James Bach http://www.satisfice.com/blog/ Elisabeth Hendrickson http://testobsessed.com/ Markus Gärtner http://www.shino.de/blog/Tuesday, June 5, 2012
  18. 18. Ressources by James Bach Topic Ressource SBTM Session Based http://satisfice.com/articles/sbtm.pdf Test Management http://people.eecs.ku.edu/~saiedian/ Exploratory Testing Teaching/Fa07/814/Resources/ exploratory-testing.pdf Hands-on Exploratory http://www.satisfice.com/tools/ procedure.pdf Testing Rapid Software Testing http://www.satisfice.com/rst.pdfTuesday, June 5, 2012
  19. 19. The basic misunderstanding • <X> is necessary but not sufficientTuesday, June 5, 2012
  20. 20. An example from agile software developmentTuesday, June 5, 2012
  21. 21. An example from agile software development Working Software over Comprehensive DocumentationTuesday, June 5, 2012
  22. 22. Derived Assumption: Working software is sufficientTuesday, June 5, 2012
  23. 23. Reality: Working software is necessary, but not sufficient!Tuesday, June 5, 2012
  24. 24. Tuesday, June 5, 2012
  25. 25. Tuesday, June 5, 2012
  26. 26. Tuesday, June 5, 2012
  27. 27. So what’s the problem? ■ Software testing, the process of verifying that a software program works as expected Source: http://en.wikipedia.org/wiki/TestTuesday, June 5, 2012
  28. 28. So what’s the problem? ... but ...Tuesday, June 5, 2012
  29. 29. So what’s the problem? ■ Test [...] Etymology From Old French test (“an earthen vessel, especially a pot in which metals were tried”) Source: http://en.wiktionary.org/wiki/testTuesday, June 5, 2012
  30. 30. What do test pilots do?Tuesday, June 5, 2012
  31. 31. What do test pilots do?Tuesday, June 5, 2012
  32. 32. What do test pilots do?Tuesday, June 5, 2012
  33. 33. What do test pilots do?Tuesday, June 5, 2012
  34. 34. What do test pilots do?Tuesday, June 5, 2012
  35. 35. Pushing the envelopeTuesday, June 5, 2012
  36. 36. Checking thingsTuesday, June 5, 2012
  37. 37. Scripted testing With a script, you miss the same things every time. Exploratory Testing @ QUEST 2008" Copyright © 2008 Cem KanerTuesday, June 5, 2012
  38. 38. Defensive programming - Check all input parameters - Ascertain availability of resources etc. Defensive programming defends against the currently impossible (c2 Wiki)Tuesday, June 5, 2012
  39. 39. Design by Contract Betrand Meyer Since 1986 Programming language: EiffelTuesday, June 5, 2012
  40. 40. Design by Contract Betrand Meyer Since 1986 Programming language: Eiffel Preconditions Postconditions InvariantsTuesday, June 5, 2012
  41. 41. TDD • Unit Tests • Black-Box Tests • White-Box Tests • RSpec etc.Tuesday, June 5, 2012
  42. 42. BDD • Acceptance Tests • (Mostly) Black-Box Tests • Based on ScenariosTuesday, June 5, 2012
  43. 43. ATDD • Bigger gap between test creation and execution • Exclusively Black Box TestingTuesday, June 5, 2012
  44. 44. An adage from agile software developmentTuesday, June 5, 2012
  45. 45. An adage from agile software development INVEST in stories and come up with SMART tasksTuesday, June 5, 2012
  46. 46. INVEST Independent Negotiable Valuable Estimatable Small TestableTuesday, June 5, 2012
  47. 47. SMART Specific Mesuarable Achievable Realistic TangibleTuesday, June 5, 2012
  48. 48. Independent Specific Negotiable Mesuarable Valuable Achievable Estimatable Realistic Small Tangible TestableTuesday, June 5, 2012
  49. 49. Acceptance Criteria • To accomodate pre-conditions Acceptance Criteria can be expressed as • Given [Precondition] When [Actor and Action] Then [Observable Result] • Naresh Jain Slide 29Tuesday, June 5, 2012
  50. 50. Specification by Example ... It is the result of a research on how teams all over the world specify, develop, test and deliver the right software, without defects, in very short iterative delivery cycles. ...Tuesday, June 5, 2012
  51. 51. What’s left for the Testers?Tuesday, June 5, 2012
  52. 52. DbC vs. TDD Design by Contract implies automated defensive programming pre- and postconditions and Test Driven Design is invariants are checked at mostly applied during design- runtime time. Expensive checks are Tests are executed whenever inhibitive appropriate - but usually not in the production environment Expensive checks are possible but unwarrantedTuesday, June 5, 2012
  53. 53. Checking vs. Testing Checking is something that we do with the motivation of confirming existing beliefs Testing is something that we do with the motivation of finding new information. http://www.developsense.com/blog/2009/08/testing-vs-checking/Tuesday, June 5, 2012
  54. 54. Test Driven Development is an oxymoron Check Driven Development?Tuesday, June 5, 2012
  55. 55. The Test-CyclesTuesday, June 5, 2012
  56. 56. Red-Green-Refactor Failing Test Simplify Coding Remove RedundancyTuesday, June 5, 2012
  57. 57. Exploratory Testing From: Paul Carvalho http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.htmlTuesday, June 5, 2012
  58. 58. So what is Exploratory Testing? "Exploratory Testing is simultaneously learning about the system while designing and executing tests, using feedback from the last test to inform the next." Elisabeth Hendrickson, @testobsessed Quoted by Paul CarvalhoTuesday, June 5, 2012
  59. 59. Cone of uncertainty Agile and Iterative Development: A Managers Guide, Craig Larman 2003Tuesday, June 5, 2012
  60. 60. Cone of uncertaintyTuesday, June 5, 2012
  61. 61. Cone of test-cases http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.htmlTuesday, June 5, 2012
  62. 62. Agile testing quadrants http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/Tuesday, June 5, 2012
  63. 63. The Prime Directive Be Thoughtful and Methodical Throughout the test procedure, as you complete the tasks, you have lots of freedom about how you do the work. But you must work methodically, and follow the procedure. In the course of creating the result for each task, you’ll find that you have to make a lot of guesses, and some of them will be wrong. But you must think. If you find yourself making wild and uneducated guesses about how the product works, areas of instability, or anything else, stop and talk to the Test Manager. James Bach, 08/26/99, http://www.satisfice.com/tools/procedure.pdfTuesday, June 5, 2012
  64. 64. Scientific Method Formulate a question Hypothesis Prediction Test AnalysisTuesday, June 5, 2012
  65. 65. Scientific Method Formulate a question Hypothesis Prediction Test AnalysisTuesday, June 5, 2012
  66. 66. Exploratory Testing Complete these five tasks: ✓ Identify the purpose of the product. ✓ Identify functions. ✓ Identify areas of potential instability. ✓ Test each function and record problems. ✓ Design and record a consistency verification test. Source: http://www.satisfice.com/tools/procedure.pdf pg. 11 to 22Tuesday, June 5, 2012
  67. 67. Identify the purpose of the product • With a tangible result (i.e. Document) • Value for others - verifiable etc. • Value for the tester - builds understandingTuesday, June 5, 2012
  68. 68. Identify functions • By actually using (skimming) the product • Explorative (not based on documentation) • Tangible output (Function outline)Tuesday, June 5, 2012
  69. 69. From James Bach: http://www.satisfice.com/tools/procedure.pdfTuesday, June 5, 2012
  70. 70. Identify areas of potential instability • Results: • Documented areas (before testing) • Documented data for testing • Documented StrategiesTuesday, June 5, 2012
  71. 71. Test each function and record problems • Actually this means: Try to do things the development teams did not think of. • Remember: A Problem is just the difference between the expected behavior and the observed behavior - whether it is cause for action has to be decided by the customerTuesday, June 5, 2012
  72. 72. Design and record a consistency verification test • Try to extrapolate if the Software would run in (slightly) modified environments. (The original 5 steps are from 1999 (gasp) and relate to specific windows versions - the basic requirements still hold through)Tuesday, June 5, 2012
  73. 73. Tool Support Area Tool URL http:// Recording spector www.spectorsoft.com/ http:// Recording BBTestAssistant www.bbsoftware.co.uk/ products.aspx http:// Recording Wink www.debugmode.com/ wink/ http://www.sirius- All in One TestExplorer sqa.com/ http://www.satisfice.com/blog/archives/77Tuesday, June 5, 2012
  74. 74. Session Based Test Management • Original description of ET : 1999 • Today : 2012 • SBTM: A way to organize exploratory testing • Working in 45 to 90 min Session based on Charters • See: http://satisfice.com/articles/sbtm.pdfTuesday, June 5, 2012
  75. 75. What are Charters Counterpart to the use case / user story Some of the same attributes - Roles, activities, actors, systems Examples: - “Analyze <Product>’s View menu functionality and report on areas of potential risk.”Tuesday, June 5, 2012
  76. 76. HierarchyTuesday, June 5, 2012
  77. 77. Hierarchy Ad-Hoc TestingTuesday, June 5, 2012
  78. 78. Hierarchy Exploratory Testing Ad-Hoc TestingTuesday, June 5, 2012
  79. 79. Hierarchy Session Based Testing Exploratory Testing Ad-Hoc TestingTuesday, June 5, 2012
  80. 80. Hierarchy Session Based Test Management Session Based Testing Exploratory Testing Ad-Hoc TestingTuesday, June 5, 2012
  81. 81. ET-SpectrumTuesday, June 5, 2012
  82. 82. ET-Spectrum Exploratory Testing according to: http://www.satisfice.com/rst.pdfTuesday, June 5, 2012
  83. 83. Can we prove it? Testing shows the presence, not the absence of bugs J.N. Buxton and B. Randell, eds, Software Engineering Techniques, April 1970, p. 16. Report on a conference sponsored by the NATO Science Committee, Rome, Italy, 27–31 October 1969. via http://en.wikiquote.org/wiki/Edsger_W._Dijkstra Program testing can be used to show the presence of bugs, but never to show their absence! Source: Notes On Structured Programming, 1972, at the end of section 3, On The Edsger Wybe Dijkstra Reliability of Mechanisms. EWD249 (1970).Tuesday, June 5, 2012
  84. 84. TDD • Technique to drive the design of your code y, t! r n sa cie s fi ce uf e s N ot t n BuTuesday, June 5, 2012
  85. 85. BDD • Technique to capture requirements and drive desired functionality y, t! r n sa cie s fi ce uf e s N ot t n BuTuesday, June 5, 2012
  86. 86. ET • Technique to drive the discovery of unwanted, unexpected and unpredicted behaviour y, t! r n sa cie s fi ce uf e s N ot t n BuTuesday, June 5, 2012
  87. 87. About me “Consulting on software development processes and software architecture since the last millennium” - @MMahlberg Google me: “Michael Mahlberg”Tuesday, June 5, 2012
  88. 88. Discussion?Tuesday, June 5, 2012
  89. 89. Flight-Envelope Photo Credits http://en.wikipedia.org/wiki/File:PerformanceEnvelope.gif Testpilots: http://www.flickr.com/photos/x-ray_delta_one/4106250812/in/photostream/ Spreading junction http://www.flickr.com/photos/elsie/4164899/in/photostream/ Tokio Tower http://www.flickr.com/photos/wilhelmja/330965980/in/photostream/ Eiffel Tower http://www.flickr.com/photos/gcattiaux/2565101112/in/photostream/ Spectrum http://www.flickr.com/photos/eiriknewth/132859486/in/photostream/ Armor http://www.flickr.com/photos/nicmcphee/1563082378/in/photostream/ Aerobatics http://www.flickr.com/photos/ssandars/6827302/in/photostream/ Airbus http://www.flickr.com/photos/mwanasimba/4097232435/Tuesday, June 5, 2012

×