Successfully reported this slideshow.

Test Automation Patterns

645 views

Published on

When implementing test automation, many people encounter problems: where to start with automation, high maintenance costs for the automated tests, or unrealistic management expectations. The good news is that solutions to these problems exist and have been effectively used by many. A “pattern” is a general reusable solution to a commonly occurring problem. Patterns have been popular in software development for many years, but they are not commonly recognized in system-level test automation. Dorothy Graham shares a collection of common problems (issues) and their solutions (patterns) which she and others are now developing as a wiki. To help resolve typical issues, Dot gives you a brief guided tour of some patterns—from Maintainable Testware and Domain-Driven Testing to Fail Gracefully and Kill the Zombies. Dot helps you recognize test automation issues and shows you how to identify appropriate patterns to help solve them.

Published in: Technology
  • Be the first to comment

Test Automation Patterns

  1. 1.       nt Session    Presented by:  Dorothy   Indepen ultant      Brought to you by:      340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 W3  Concurre 4/9/2014    10:30 AM          “Test Automation Patterns”       Graham dent Testing Cons             300, 68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 
  2. 2. Dorothy Graham Independent Test Consultant   In software testing for forty years, Dorothy Graham is coauthor of four books— Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation—and is currently working with Seretta Gamba on a new book on test automation patterns. A popular and entertaining speaker at conferences and seminars worldwide, Dot has attended STAR conferences since the first one in 1992. She was a founding member of the ISEB Software Testing Board and a member of the working party that developed the ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn more about Dot at DorothyGraham.co.uk.
  3. 3. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 1 Test Automation Patterns Prepared and presented by © Seretta Gamba, Dorothy Graham 2014 Dorothy Graham @DorothyGraham info@dorothygraham.co.uk www.DorothyGraham.co.uk 2 Contents Background Issues and examples Test automation Patterns Wiki Conclusion Test Automation Patterns
  4. 4. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 3 Origin of patterns in software – 1970s: architect Christopher Alexander describes patterns for buildings (e.g. light from 2 walls) – 1987: Kent Beck & Ward Cunningham experimented & presented at OOPSLA – 1994: “Gang of Four” wrote a book applying pattern principles to software design •  “Design Patterns: elements of reusable object-oriented software”, Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides – now widely accepted in software development community 4 What is a pattern? – a generally reusable solution to a commonly occurring problem within a given context – a description or template for how to solve a problem that can be used in many different situations – a pattern is NOT •  a finished solution that can be ‘plugged in’ or transformed directly into code •  prescriptive •  a set of steps – a pattern needs to be implemented within context
  5. 5. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 5 Where did these automation patterns come from? – Experiences book published 2012 – Seretta Gamba contributed Ch 21 •  Automation through the back door, by supporting manual testing – Seretta is a developer, used to using patterns – when she read the other chapters, she thought: “Patterns! Patterns!” – she wrote up test automation patterns as a book & asked me to join her – Word didn’t work very well, so we made a wiki 6 Introduction to the wiki •  the Test Automation Patterns wiki – ISSUES – describe problems in automation – PATTERNS – proven ways to overcome problems – other things: acknowledgements, references, some failure patterns •  in this presentation – I am using an off-line version of the wiki for demonstration purposes – later I will tell you how you can gain access to the “real” wiki
  6. 6. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 7 Test automation patterns wiki •  purpose: share information, ideas and experiences about test automation, using issues and patterns •  mind-maps summarise issues and patterns (not clickable as links) •  issues and patterns are classified into sub- sections (e.g. management, process, etc.) •  “diagnostics” help you find your way into the most relevant issues for you (new) 8 Contents Background Issues and examples Test automation Patterns Wiki Conclusion Test Automation Patterns
  7. 7. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 9 Issues (problems) •  what problems affect you in test automation? – what causes the most trouble? – what takes longer than it should? – what is holding you back in your automation? – what are you struggling with? – what would you most like to improve? 10 Test automation Issues: 4 examples •  HIGH MAINTENANCE COST –  Maintenance of the test automation scripts takes a long time and costs more than it’s worth. •  NO PREVIOUS TEST AUTOMATION –  Your first time automating, or automation has failed and you are starting again. •  STALLED AUTOMATION –  Automation has been tried, but it never “got off the ground” or has now fallen into disuse. •  UNREALISTIC EXPECTATIONS –  Management has unrealistic expectations about what test automation can & can’t do.
  8. 8. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 11 Issue: HIGH MAINTENANCE COST •  Maintenance of the test automation scripts takes a long time and costs more than it’s worth •  Questions to ask: – what cost (or time) would be acceptable? – how is the cost measured? – Note: answers may be different in different contexts 12 Issue: HIGH MAINTENANCE COST 1 •  What are your symptoms? – When software changes, e.g. window sequence is different, tests fail. More effort to edit the existing tests than to start again. (Capture/replay used) – Diagnostic question: Are the scripts structured and reusable? •  1) No, scripts are mainly “stand-alone” without much reuse, automators are “re-inventing” •  2) Yes, structured and reusable scripts, but it needs automators to add new tests, and we are short of time and/or automators visit offline wiki 
  9. 9. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 13 Issue: HIGH MAINTENANCE COST 2 •  What are your symptoms? – No time allowed to maintain the automated tests – Tests are written in the tool’s language, but the tool has changed, or we have to change tools 14 HIGH MAINTENANCE COST (Tour) - 1 •  Symptom: SUT changes -> large cost to update – MAINTAINABLE TESTWARE – Symptom: Scripts are mainly “stand-alone” without much reuse, automators are re-inventing •  GOOD PROGRAMMING PRACTICES –  DESIGN FOR REUSE, OBJECT MAP – Symptom: Structured & reusable scripts, automator / time shortage •  DOMAIN-DRIVEN TESTING –  ABSTRACTION LEVELS, KEYWORD-DRIVEN TESTING
  10. 10. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 15 HIGH MAINTENANCE COST (Tour) - 2 •  Symptom: No time to maintain automated tests (we know we should) – MANAGEMENT SUPPORT •  Explain technical debt, SELL THE BENEFITS •  Symptom: Scripts in tool language, tool changed or having to change tools – TOOL INDEPENDENCE – ABSTRACTION LEVELS 16 DOMAIN-DRIVEN TESTING •  Uses a “Domain-specific test language” – keywords from the application domain •  insurance: New Policy or Make Claim •  mobile: Make Call or Update Contacts – each keyword is processed by a script that then processes any data for that keyword – tests in spreadsheet or databases •  make it easy for business testers to add new tests •  needs technical support (to set up the TESTWARE ARCHITECTURE and when things go wrong)
  11. 11. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 17 Testware architecture: abstraction levels abstraction here: easier to write automated tests:  widely used DOMAIN-DRIVEN TESTING abstraction here: easier to maintain, and change tools:  long life GOOD PROGRAMMING PRACTICES, GOOD DEVELOPMENT PROCESS testware  architecture   Testers     Test  Execu/on  Tool   runs  scripts   High Level Keywords Structured Scripts structured   testware   Test Automator(s) write  tests  (in  DSTL)   18 Some other patterns •  FAIL GRACEFULLY – make sure a test “cleans up” after itself even if it fails (so subsequent tests can run without worry) – alternative: FRESH SETUP, where each tests re- sets its initial state, no matter what happened before, or SHARED SETUP •  KILL THE ZOMBIES – make sure tests are actually doing something; delete or archive tests no longer in use
  12. 12. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 19 Contents Background Issues and examples Test automation Patterns Wiki Conclusion Test Automation Patterns 20 About these patterns (and the wiki) •  the wiki has a limited set of patterns – and a limited “depth” for each pattern •  patterns use other patterns – it can get rather complex •  we are focusing on system-level automation rather than unit test automation – See Gerard Meszaros’ book “xUnit Test Patterns: Refactoring test code” for unit test patterns •  these are test automation patterns, not test patterns
  13. 13. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 21 Diagnostics (new) •  modeled on a doctor’s questions to a new patient – ask questions, and ask more questions, depending on the answers •  First level – new to automation? Go to NO PREVIOUS TEST AUTOMATION – completely happy? giving up (good riddance)? •  share your story – want to improve or revive your automation? •  see next level 22 Improve or revive automation •  Most pressing issue: – lack of support (from mgt, testers, dev) – lack of resources (staff, s/w, h/w, time) – lack of direction (what to aut, what architecture) – lack of specific knowledge (of SUT, tool, maintainability) – unsatisfactory quality of automation (ROI, maintenance, unreliable) •  each takes you to more detail in the wiki, and points you to an issue with resolving patterns
  14. 14. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 23 Lack of support •  different kinds of support may be lacking – managers pay “lip service” – don’t see the value – managers have UNREALISTIC EXPECTATIONS – testers don’t help the automators – developers don’t help the automators – specialists don’t help the automators (DB, networks) – no one helps the newbies 24 Contents Background Issues and examples Test automation Patterns Wiki Conclusion Test Automation Patterns
  15. 15. info@dorothygraham.co.uk srttgmb@yahoo.com © Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk 25 Conclusion •  we hope our wiki will help you identify and understand problems (issues) and give you ideas (patterns) for dealing with them •  the wiki is free, by invitation – email me at info@DorothyGraham.co.uk for an invitation – if you already have a wikispaces account, also give me your wiki user name – an invitation will be sent from the wiki giving you instructions for how to set up your user name & password 26 Finally •  Thank you for coming today •  Any questions? •  Contact us: – info@dorothygraham.co.uk – srttgmb@yahoo.com •  We welcome your feedback  •  We welcome your stories & experiences •  All the best with your test automation!

×