Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automating Strategically or Tactically when Testing


Published on

"Test Automation" can be viewed as strategic or tactical.
This presentation describes reasons for making this distinction and how you know if you are working strategically or tactically when you automate as part of your test approach.

Published in: Software
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Automating Strategically or Tactically when Testing

  1. 1. Automating Tactically and Strategically Alan Richardson 4 4 4 @eviltester TESTING ASSEMBLY 09/2017, Helsinki
  2. 2. Do you automate tactically or strategically?
  3. 3. Automating Tactically allows us to move fast, and target immediate needs
  4. 4. Automating Strategically allows us to gain budget and support for longer periods of work
  5. 5. We may believe we are automating strategically but are actually automating tactically
  6. 6. Combine tactical and strategic approaches to better our testing efforts.
  7. 7. Secrets of Successful Practical Automating 4 Get work done 4 Automate Flows 4 Multiple Abstractions 4 Abstract to libraries, not frameworks
  8. 8. Testing field argues about 4 Testing vs Checking 4 You can't automate testing "Test Automation" is contentious
  9. 9. An Example "Is what follows a Test?" ...
  10. 10. @Test public void createPreviewMVP() throws IOException { String args[] = { new File(System.getProperty("user.dir"), "").getAbsolutePath()}; new PandocifierCLI().main(args); String propertyFileName = args[0]; File propertyFile = new File(propertyFileName); PandocifierConfig config; config = new PandocifierConfigFileReader(). fromPropertyFile(propertyFile); Assert.assertTrue(new File( config.getPreviewFileName()).exists()); }
  11. 11. An Example "Ce n'est pas un test", ... 4 a Test, it says it is a @Test 4 an Automated Test, it checks something (file exists) 4 an Automated Test, it Asserts on the check 4 a Tool - I use it to automatically build one of my books 4 not an Application - it doesn't have a GUI or a standalone execution build
  12. 12. This is automated execution of a process
  13. 13. We Automate Parts of our Development Process: Coding, Checking, Asserting, ...
  14. 14. Tactical Reality - I don't really care, I just want to get work done Suggests we also have 'strategic reality'
  15. 15. Tools vs Abstractions 4 Gherkin 4 Specflow, Cucumber - - - - - - - - - - - - - - 4 Gherkin is a DSL templating language 4 Specflow, Cucumber are template parsers 4 We have to create abstractions to fill in the gap
  16. 16. E. W. Djkstra on Abstraction "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." 4 1972 ACM Turing Lecture: 'The Humble Programmer'
  17. 17. Regression Testing The system rarely actually suffers "Regression"
  18. 18. Continuous Condition Assertion 4 TDD, ATDD, Automated execution 4 All of these provide an ongoing safety net which might mean we don't need to revisit areas in the future (if we also tested them) 4 They don't mean that we have 'tested' them now
  19. 19. Coverage - All coverage is Model Based 4 Code coverage based on source code model 4 What about requirements, system interfaces, states, flows? 4 Code coverage is a useful metric for TDD coverage 4 coverage models based on risk
  20. 20. Tactical vs Strategic Automating Strategic does not mean 'strategy document'
  21. 21. Automating as part of a Testing Strategy
  22. 22. Automating Tactically vs Strategically What is Tactical? What is Strategic? Solve a problem Agreed & Understood Goals Short Term Long Term Change as necessary Slower to Change Your Time 'project' time
  23. 23. Warning - sometimes tactics look like strategy 4 Use of Jenkins is not Continuous Integration 4 Cucumber/Specflow (Gherkin) != ATDD or BDD 4 Automated Deployments != Continuous Delivery 4 Containers != always good environment provision 4 Page Objects != always good system abstractions
  24. 24. You know you are Automating strategically when... 4 Reduced Maintenance & Change 4 Everyone agrees why 4 No fighting for 'time' 4 It's not about roles 4 It gets done
  25. 25. How do we automate? 4 not about "test automation" 4 it is about automating tasks 4 automate the assertions used to check that the stories are still 'done'
  26. 26. Automate flows through the system - vary data - abstract the execution
  27. 27. Automating Strategically with Abstractions see also "Domain Driven Design" "modeling the system in code" 4 abstractions are about modelling - people seem to be scared of having too many models
  28. 28. No Abstractions - Web Testing Example
  29. 29. @Test public void canCreateAToDoWithNoAbstraction(){ WebDriver driver = new FirefoxDriver(); driver.get(""); int originalNumberOfTodos = driver.findElements( By.cssSelector("ul#todo-list li")).size(); WebElement createTodo = driver.findElement("new-todo"));; createTodo.sendKeys("new task"); createTodo.sendKeys(Keys.ENTER); assertThat(driver.findElement("filters")).isDisplayed(), is(true)); int newToDos = driver.findElements( By.cssSelector("ul#todo-list li")).size(); assertThat(newToDos, greaterThan(originalNumberOfTodos)); }
  30. 30. But there were Abstractions (Lots of them) 4 WebDriver - abstration of browser 4 FirefoxDriver - abstraction Firefox (e.g. no version) 4 WebElement - abstraction of a dom element 4 createToDo - variable - named for clarity 4 Locator Abstractions via methods e.g. findElement 4 Manipulation Abstractions .sendKeys 4 Locator Strategies
  31. 31. Using Abstractions - Same flow
  32. 32. @Test public void canCreateAToDoWithAbstraction(){ TodoMVCUser user = new TodoMVCUser(driver, new TodoMVCSite()); user.opensApplication().and().createNewToDo("new task"); ApplicationPageFunctional page = new ApplicationPageFunctional(driver, new TodoMVCSite()); assertThat(page.getCountOfTodoDoItems(), is(1)); assertThat(page.isFooterVisible(), is(true)); }
  33. 33. What Abstractions were there? 4 TodoMVCUser 4 User 4 Intent/Action Based 4 High Level 4 ApplicationPageFunctional 4 Structural 4 Models the rendering of the application page
  34. 34. REST Test - No Abstractions
  35. 35. @Test public void aUserCanAccessWithBasicAuthHeader(){ given(). contentType("text/xml"). auth().preemptive().basic( TestEnvDefaults.getAdminUserName(), TestEnvDefaults.getAdminUserPassword()). expect(). statusCode(200). when(). get(TestEnvDefaults.getURL().toExternalForm() + TracksApiEndPoints.todos); }
  36. 36. But there were Abstractions 4 RESTAssured - given, when, then, expect, auth, etc. 4 RESTAssured is an abstraction layer 4 Environment abstractions i.e. TestEnvDefaults Lots of Abstractions But a lack of flexibility
  37. 37. Different abstractions to support flexibility
  38. 38. @Test public void aUserCanAuthenticateAndUseAPI(){ HttpMessageSender http = new HttpMessageSender( TestEnvDefaults.getURL()); http.basicAuth( TestEnvDefaults.getAdminUserName(), TestEnvDefaults.getAdminUserPassword()); Response response = http.getResponseFrom( TracksApiEndPoints.todos); Assert.assertEquals(200, response.getStatusCode()); }
  39. 39. An API Abstraction
  40. 40. @Test public void aUserCanAuthenticateAndUseAPI(){ TodosApi api = new TodosApi( TestTestEnvDefaults.getURL(), TestEnvDefaults.getUser()); Response response = api.getTodos(); Assert.assertEquals(200, response.getStatusCode()); }
  41. 41. Automating API vs GUI
  42. 42. Automate at the interfaces where you can interact with the system.
  43. 43. Architecting to Support... - Testing - Support - Deployment - Automating - Monitoring - Devops - Releasing - etc.
  44. 44. Abstractions support different types of testing 4 regular automated execution with assertions 4 creation and maintenance by many roles 4 exploratory testing 4 stress/performance testing (bot example) requires thread safe abstractions
  45. 45. Good abstractions encourage creativity, they do not force compliance and restrict flexibility.
  46. 46. Supports Law of Requisite Variety "only variety can destroy variety" - W. Ross Ashby "only variety can absorb variety" - Stafford Beer
  47. 47. Frameworks can 'Destroy' variety 4 Frameworks force us to comply with their structure 4 or things get 'hard' 4 or we don't reap the benefits from the Framework 4 Some things are not possible 4 Some things never occur to you to do
  48. 48. We have a lot of models 4 user intent model - what I want to achieve 4 user action model - how I do that 4 interface messaging model - how the system implements that 4 admin models, support models, GUI models
  49. 49. And models overlap e.g. user can use the API or the GUI to do the same things
  50. 50. When we don't do this 4 test code takes a long time to maintain 4 test code takes too long to write 4 test code is hard to understand 4 test code is brittle - "every time we change the app the tests break/ we have to change test code" 4 execution is flaky - intermittent test runs - "try running the tests again"
  51. 51. Example: Tactical Approach on Agile projects 4 exploratory testing to get the story to done 4 create a tech debt backlog for 'missing' automated tests 4 @Test code is tactical and we can knock it up (not as good as production)
  52. 52. Example: Strategic Approach on Agile Projects 4 automate assertion checking of acceptance criteria for "done" 4 ongoing execution of the assertions for the life of the story
  53. 53. Example: Strategic Approach on Agile Projects implies... 4 maintained for the life of the story in the application 4 deleting the @Test code when no longer relevant 4 amending code structure to make relevant as stories change and evolve 4 @Test code is strategic asset (just like production code)
  54. 54. Why labour the point? 4 "test automation" often means "testers do the automating" 4 testers may not program as well as programmers 4 testers may not know various programming patterns 4 when programmers see @Test code written by testers they often don't want to touch it or maintain it
  55. 55. Move from Abstractions such as 'Programming' and 'Testing' to 'Developing'
  56. 56. I say I'm a "Test Consultant" I'm really a "Software Development Consultant" Testing is Part of the Development process
  57. 57. Summary 4 Automate Strategically 4 ongoing assertion checking automated in @Test code 4 Abstractions model the system in code 4 Write abstractions at multiple levels 4 Good abstractions can support exploratory testing 4 Write good code all the time
  58. 58. Learn to "Be Evil" 4 4 @eviltester 4
  59. 59. Learn About Alan Richardson 4 4
  60. 60. Follow 4 Linkedin - @eviltester 4 Twitter - @eviltester 4 Instagram - @eviltester 4 Facebook - @eviltester 4 Youtube - EvilTesterVideos 4 Pinterest - @eviltester 4 Github - @eviltester
  61. 61. BIO Alan is a test consultant who enjoys testing at a technical level using techniques from psychotherapy and computer science. Alan is the author of the books "Dear Evil Tester", "Java For Testers" and "Automating and Testing a REST API". Alan's main website is and he blogs at (see also