Get the Balance Right: Acceptance Test Driven Development, GUI Automation and Exploratory Testing


Published on

My presentation to support my paper (same title) from PNSQC 2012.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • “by repeatedly running the same tests, we will not only not find new bugs, but thebugs that are left will be more resistant to the tests we do perform [Beizer, 1990].
  • Developers have an exploratory mindset, but at the moment, ATDD explores to discover the real requirements. TDD is not exploratory in the same way. There is an aspect that is exploratory, the first TDD test will lead you to the nest test, the small tests lead you to TDD.
  • .
  • Get the Balance Right: Acceptance Test Driven Development, GUI Automation and Exploratory Testing

    2. 2. MICHAEL LARSENSenior Tester, Sidereel.comChair, Education Special InterestGroup, Association For Software TestingBlack Belt, Miagi-do School of Software TestingFounder/Facilitator, Weekend Testing AmericasTESTHEAD: http://mkltesthead.comTwitter: @mkltesthead
    3. 3. WHAT’S HAPPENEDSINCE 2001?Eleven years since the Agile ManifestoTest Driven Development (TDD) has emerged as astandard to guide code development driven by tests.Acceptance Test Driven Development (ATDD)extends model.Proliferation of automated testing tools for a varietyof platforms and needs.
    4. 4. DO WE STILL NEEDTESTERS?NO: Testing is dead. With the developmentof these tools, developers now do thetesting earlier and up front [Savioa, 2011].YES: Apply reason and dynamic thought toproblems. Thinking, actively engagedtesters are needed now more than ever[Tomlinson, 2011].
    5. 5. TDD, WHAT IS IT?
    6. 6. ATDD, WHAT IS IT?Places the focus on User Stories from theexpectations of the customers.Is geared towards the members of the teamthat are not necessarily programmers.Uses tools that allow a more naturallanguage to present the requirements.
    7. 7. ATDDScenario: Twitter user signs in Given I connect to twitter from the sign in page When I log into twitter with good credentials Then I go to the user private profile page And I can confirm my twitter credentialsExpressed in Gherkin, the language usedwith Cucumber.Tied to actual coded methods to make theprocess work.
    8. 8. TDD AND ATDD ARENOT TESTING!TDD and ATDD are design processes.Help develop clean, well functioning code.Help ensure software being developedmeets needs of the project
    9. 9. WHY ISN’T THISTESTING?Testing:“The art and the act of asking questions ofa product, then developing & devising newand more inventive questions based on theanswers we receive.”
    10. 10. WHERE DO TDD ANDATDD EXCEL?Disciplined approach to developing cleanersoftware.Act as a brake on "cowboy code".New programmers can see code in contextwith the tests that have been written.
    11. 11. WHERE DO TDD ANDATDD EXCEL?TDD and ATDD work well with Continuous Integration.Keeps issues based on dependencies to a minimum.Programmers focus on "just enough design" and "justenough development".Entire product team does design and developmentwork.
    12. 12. WHERE DO TDD ANDATDD FALL SHORT?Unrealistic expectations.Only be as good as the overall skill of thedeveloper(s).Can be over applied.
    13. 13. WHAT IS FRONT ENDGUI AUTOMATION?Proliferation of tools that help simulate the actionsof users.Follow standard workflows • (logging in, navigating to pages, clicking on links, filling in forms, etc.).Tools range from lightweight apps like Texter up tofull feature development suites(Selenium/WebDriver, FitNesse, TestComplete, etc.).
    14. 14. HOW DOES GUIAUTOMATION DIFFERFROM TDD/ATDD?Helps with constructingacceptance test casesHelps to check anddemonstrate the acceptancecriteria has been met.
    15. 15. GUI AUTOMATIONRemember our original example:
    16. 16. GUI AUTOMATIONCucumber statements = abstraction of realcode.Map to appropriate tools that make statementswork on the application or browser.
    17. 17. GUI AUTOMATIONThe code below is Ruby, using Capybara tomake calls to the browser:
    18. 18. GUI AUTOMATIONStep Definitions -> Capybara and Ruby
    19. 19. WHERE DOES GUIAUTOMATION EXCEL?Tremendous blessing when dealingwith repetitious steps (set up and takedown).Important part of Acceptance Testverification.Semi proxy for human interaction.
    20. 20. GUI AUTOMATIONDEFICIENCIESGUI tends to be the most fragilelayer of an application.Tremendous overhead of scriptmaintenance.Not good at checking context.
    22. 22. WHAT IS EXPLORATORYTESTING?“Scientific thinking in real-time” [Bach, 2012]Puts test design and test executiontogether.The tests that we did before inform the teststhat we will do now. Those tests will informthe tests we will perform later.
    23. 23. EXPLORATORYTESTINGConcepts are defined, but they are not completelypredefined or run in a rigid sequence.Allows for a development of test ideas that areinteresting.Example: Automated tests can be madeexploratory by randomizing order.Purpose: Discover where dependencies or stateconditions do exist.
    24. 24. EXPLORATORY TESTINGVS. AUTOMATEDTESTINGInstead of saying "automated testing", Iprefer to use the term "computer aidedtesting".Exploratory Testing can, and oftendoes, use elements of computer aidedtesting to help accomplish its goals.
    25. 25. AUTOMATION CAN’SAND CANNOT’SAutomation Can: Automation Cannot:Generate test data to be used in Create curiosity.forms or as variable values.Parse the output of a program and Make sapient (actively thinking)use it as input for another program. decisions.Create a log of actions and Invent a new idea or an approachtransactions. based on the output of a sequence of tests.Alert if an assertion is met or not Notice something unexpectedmet. (unless we have already defined what is unexpected).Search for patterns in output, or Make a judgment call as to thehelp to reveal patterns we did not value or importance of a piece ofknow about. functionality.
    26. 26. WHERE DOESEXPLORATORY TESTINGEXCEL?Allows the tester to choose the sequence ofsteps that can provide answers about thestate or condition of a product.Imagination and creativity open uppossibilities.When there is little in the way of formalizeddocumentation.
    27. 27. EXPLORINGREQUIREMENTSRemember this acceptance test?Scenario: Twitter user signs in through Twitter Given I connect to twitter from the sign in page When I log into twitter with good credentials Then I go to the user private profile page And I can confirm my twitter credentials
    28. 28. SAMPLE EXPLORATIONQUESTIONSIs there a login interface?Does it provide for proper error handling if I cant login?Does it give me feedback to let me know that I havesuccessfully logged in?Can I trick the system into letting me log in withimproper credentials?Does it present me with information that could help meguess a login without actually having one?Where else could I use this functionality?
    29. 29. EXPLORATORY TESTINGIS PART OF TDD/ATDDProposing a failing test first, and thencreating code that meets the acceptancecriteriaProgrammer must consider what theapplication needs to doDetermine which avenue(s) to implementthe feature(s).
    30. 30. BENEFITS OFEXPLORATORY TESTINGWalking down various pathsJumping off point to look in other placesSimple Cucumber statement "And let me see that":And /^let me see that$/ do puts "PAUSED - Press Enter to continue: #{$1}aaa" puts "" $stdin.getsEnd
    31. 31. EXPLORATORYTESTING’S DEFICIENCIESLess likely to find new revelations by goingover the same ground (pesticide paradox)Not enough time to go through allpossibilities.Sentient humans do it. Sentient humans getbored. • SBTM techniques can help
    32. 32. CAN WE CREATE ABALANCE?TDD, ATDD, and front end GUI automationare processes.Exploratory Testing is a process and amindset.Developing and defining acceptance isexploratory.Front End GUI Automation starts fromcapturing discoveries made whileexamining requirements.
    33. 33. THE ROLE OFTESTERS”Testing is dead" == testing needed to be apart of an entire process of quality andimprovement, starting at the earliest stagesof design and development.Exploration, automation, and testing mustcome earlier in the process, and involve notjust testers, but everyone on the productteam.
    34. 34. STRIKING A BALANCECan we leverage the "testingeverywhere" idea, and help developa balance for all of these goals?Can we see them asinterdependent, and not as separateand standalone activities?
    35. 35. TDD AND ATDD TAKE ANEXPLORATORY MINDSET • Testing is part of the initial development effort. • Testing comes first. • Putting the emphasis on TDD/ATDD makes sure that it meets the criteria it is being designed for.Testers can also help guide and provide consideration fortestability in areas the programmers might not consider.Asking "what if" questions vital at this stage
    36. 36. FOCUS ON DELIVERINGBUSINESS VALUEQuestion 1: "is what I am doing helpingto deliver real value to my customers"?If we put the business value first, thatwill guide us in our testing efforts andthe approaches that we use[Crispin, 2008].
    37. 37. AUTOMATION DOES NOTHAVE TO BE PERMANENTMake room for "throwaway automation”; stepsthat can be used to help explore differentavenues.Freight or passenger trains vs. taxi cabs. • Freight and passenger trains limited to the rails the trains can ride on. • Taxi cab can be called to appear anywhere and take us anywhere.
    38. 38. EXPLORATION IS AMINDSETExploratory Testing is a way of thinkingabout questions and answers.Exploratory Testing does not live outsideof automated testing.Automation and Computer Aided Testingcan be central to exploration.
    39. 39. ZEN MIND,BEGINNERS MINDDomain Expert Level Exploratory TestingNovice Level Exploratory TestingThey are different.Source of mental friction is knowing theeccentricities of your application and whenthey can get in the way of effective testing andexploration.
    40. 40. USE PERSONAS TOVISUALIZE GOALSPut yourself into the shoes of as many potentialcustomers of our product as I can.That means understanding the interests, desiresand ways of interaction of many different groups ofpeople.The ways that they interact with the devices theyuse to discover content/interact is very different.
    41. 41. SESSION BASEDTECHNIQUES• Developing small charters and executing on them in set periods of time.• Focus testing areas that are most important.• Focus on a small sub system.Keep our testing fresh, our eyes and mindsfocused, work on whats most important.
    42. 42. CONCLUSIONTDD focuses on writing correct and clean code.ATDD focuses on making sure the functionalitybeing delivered makes good on the goals andpromises made to our customers.Automation happens at many levels.Exploration happens at all of these levels, and isan approach and a mindset, not a methodology.Working together, the odds of quality software aremuch more heavily in our favor.
    43. 43. QUESTIONS ???