Zen and the Art of Automated Acceptance Test Suite Maintenance

4,889 views
4,675 views

Published on

The benefits of BDD (Behaviour-Driven Development)-style automated acceptance tests are huge. Far beyond simply testing your application, BDD uses automated acceptance tests to improve team collaboration and communication, focus development efforts on truly valuable features, and provide meaningful progress reports and reliable feature documentation.

However one of the biggest challenges to implementing Automated Acceptance Testing is writing them in a way that will be easy to maintain as the project progresses. Indeed, the cost of maintaining the acceptance test suite should not be more than the value that it provides.

This talk explores strategies for writing maintainable and meaningful automated acceptance tests, including aspects such as:
Challenges to maintaining automated acceptance tests
How to organise and structure your tests more effectively
Writing truly meaningful acceptance tests
When to test the UI, and when to test the backend
How to deal with database setup and teardown
How to avoid test fragility
How to get the most out of ATDD reporting

Published in: Technology

Zen and the Art of Automated Acceptance Test Suite Maintenance

  1. 1. and the art ofAutomated Acceptance Test Suite MaintenanceAn Inquiry into Test Automation QualityZEN
  2. 2. John Ferguson SmartConsultantTrainerMentorAuthorSpeakerCoder
  3. 3. Automated Acceptance Testing is valuable
  4. 4. Faster release cycles
  5. 5. Improved visibility
  6. 6. Reduce risk and uncertainty
  7. 7. Automated Acceptance Testing is not easy
  8. 8. Fragile?
  9. 9. Expensive to maintain?
  10. 10. Slow to run?
  11. 11. Is it worth it?
  12. 12. “It is fun to have funBut you have to know how”Sure it is!
  13. 13. Three rules of writingmaintainable acceptance testsA test should only need changing if the subject ofthe test changesInvolve the testers earlyReport on what requirements you verified,not what tests you ran
  14. 14. Rule #1 - StabilityA test should only changeif the concept under test changes
  15. 15. What do ogres and good automated tests have in common?
  16. 16. (This test is not like an ogre)
  17. 17. The Record-Replay Scamin theoryin practice
  18. 18. Remember...good automated acceptance tests have layers
  19. 19. Start off with the business objectivesBusiness-­‐level  requirements
  20. 20. Describe the business flowHigh-­‐level  steps  (business  flow)
  21. 21. Describe the page interactionsPage-­‐level  interac<ons
  22. 22. Page implementation detailsHTML  details  go  here
  23. 23. Business  RulesBusiness  FlowPage/Component  interac<onsPage/Component  detailsGood automated acceptance tests have layers
  24. 24. Business  RulesBusiness  FlowPage/Component  interac<onsPage/Component  detailsGood automated acceptance tests have layersOnly  expose  what  the  business  needs  to  knowScenario: Register for online bankingGiven that Joe wants to register for online bankingWhen he goes to the registration pageAnd he enters Joe in the first name fieldAnd he enters Bloggs in the surname fieldAnd he enters Joe@bloggs.com" in the email fieldAnd he enters 01/01/1980 in the date of birth fieldAnd he enters 1 George street in the street fieldAnd he enters Sydney in the city fieldAnd he enters 2000 in the post code fieldAnd he clicks on submitThen his application should be created in a pending stateAnd he should be sent a PDF contract to sign by emailWho cares!
  25. 25. Business  RulesBusiness  FlowPage/Component  interac<onsPage/Component  detailsGood automated acceptance tests have layersOnly  expose  what  the  business  needs  to  knowScenario: Register for online bankingGiven that Joe wants to register for online bankingWhen he submits his application onlineThen his application should be created in a pending stateAnd he should be sent a PDF contract to sign by email
  26. 26. Business  RulesBusiness  FlowPage/Component  interac<onsPage/Component  detailsGood automated acceptance tests have layersCentralize  and  reuse  test  logic  Centralize  and  reuse  test  implementa<on  details
  27. 27. Business  RulesBusiness  FlowPage/Component  interac<onsPage/Component  detailsGood automated acceptance tests have layersUse  UI  tests  for  tes<ng  user  interac<ons  with  the  screensNon-­‐UI  tests  are  OK  for  tes<ng  business  logic  and  calcula<ons
  28. 28. How much should I test?When  in  doubt,  go  broad  and  shallowDeep  and  narrow  leads  to  waste
  29. 29. Rule #2 - RelevanceReport what requirements you verified,not what tests you ran
  30. 30. Feature CoverageHow many requirements areyou really testing?
  31. 31. RequirementsCapabilities/EpicsBook  flights  onlineManage  Frequent  Flyer  accountFeaturesSearch  available  flightsBook  flight  with  pointsPay  with  credit  cardView  account  statusReal  <me  no<fica<onsStatus  upgradesAcceptance Criteria-­‐  no<fica<ons  for  delayed  flights-­‐  no<fica<on  for  cancelled  flights-­‐  no<fica<on  of  baggage  carousel-­‐  Should  see  current  points-­‐  Should  see  special  offers-­‐  should  see  current  statusAutomated TestsGiven  Joe  is  booked  on  flight  QF108And  Joe  is  registered  for  SMS  no<fica<onsWhen  the  flight  is  delayedthen  Joe  should  be  sent  an  SMS  no<fica<onGiven  Joe  is  booked  on  flight  QF108And  Joe  is  registered  for  email  no<fica<onsWhen  the  flight  is  cancelledthen  Joe  should  be  sent  an  email  no<fica<on  with  an  alterna<ve  flightGiven  Joe  is  booked  on  flight  QF108And  Joe  is  registered  for  email  no<fica<onsWhen  the  flight  is  delayedthen  Joe  should  be  sent  an  email  no<fica<on? ?? ?Given  Joe  is  a  <status>  Frequent  Flyer  with  <current-­‐points>  pointsAnd  he  has  earned  <points>  points  this  monthThen  his  new  status  should  be  <new-­‐status>status    |  current-­‐points  |  points  |  new-­‐statusSLIVER  |  9800                                        |  100          |  SILVERSLIVER  |  9900                                        |  100          |  GOLDGOLD    |  49900                                      |  100          |  PLATINUM
  32. 32. How  many  failedTest  results  can  tell  us...How  many  tests  passedHow  many  weren’t  run
  33. 33. Test  results  can  tell  us...What  tests  exist  for  a  given  featureHow  stable  the  feature  is
  34. 34. Test  results  can  tell  us...How  a  feature  was  tested
  35. 35. Test  results  can  tell  us...What  a  feature  looks  like
  36. 36. Test  results  can’t  tell  us  what  features  were  not  tested?
  37. 37. RequirementsCapabilities/EpicsBook  flights  onlineManage  Frequent  Flyer  FeaturesSearch  available  Book  flight  with  pointsPay  with  credit  cardView  account  Real  <me  no<fica<onStatus  upgradesAcceptance Criteria-­‐  no<fica<ons  for  delayed  flights-­‐  no<fica<on  for  cancelled  flights-­‐  no<fica<on  of  baggage  carousel-­‐  Should  see  current  points-­‐  Should  see  special  offers-­‐  should  see  current  statusAcceptance  criteria  defined  and  tests  workingAcceptance  criteria  defined,  but  no  testsAcceptance  criteria  defined,  tests  specified  but  not  implementedFailing  testsNo  acceptance  criteria  definedTest ResultsFeature Coverage
  38. 38. RequirementsCapabilities/EpicsBook  flights  onlineManage  Frequent  Flyer  FeaturesSearch  available  Book  flight  with  pointsPay  with  credit  cardView  account  Real  <me  no<fica<onStatus  upgradesAcceptance Criteria-­‐  no<fica<ons  for  delayed  flights-­‐  no<fica<on  for  cancelled  flights-­‐  no<fica<on  of  baggage  carousel-­‐  Should  see  current  points-­‐  Should  see  special  offers-­‐  should  see  current  statusDoneNot  tested  at  allTests  not  finished  yetBrokenNothing  done  yetTest ResultsFeature Coverage
  39. 39. Feature CoverageHigh  level  capabili<es
  40. 40. Feature CoverageWhat  features  are  defined  for  this  capability?
  41. 41. Feature CoverageWhat  stories  are  defined  for  this  feature?How  many  stories  have  automated  acceptance  criteria?
  42. 42. Feature CoverageWhat  stories  are  defined  for  this  feature?How  many  stories  have  automated  acceptance  criteria?What  acceptance  criteria  have  been  automated?
  43. 43. Rule #3 - Involve the testers earlyThe most meaningful tests are writtenbefore the application is built
  44. 44. Storybug  reportsWorking  code boring  manual  tes<ngWASTEHow do you builda feature?
  45. 45. Working  code  and  Working  Automated  Acceptance  TestsExploratory  tes<ng,  usability  tes<ng...Shared  understandingStoryExamplesAutomated  acceptance  criteriaHow do you builda feature?
  46. 46. Good  automated  acceptance  tests  have  layersInvolve  the  testers  earlyFeature  Coverage  tells  you  where  you  are  atTakeaways
  47. 47. To learn more...h[p://thucydides.info
  48. 48. Thank youJohn  Ferguson  Smart

×