Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Matthew Attaway, Perforce Software


Published on

Transitioning to agile development isn't any easier for the test team than it is for the development team. New features and fixes appear daily, priorities change fluidly as customer feedback shapes the backlog, and design documents are frequently sparse as developers use iterations to zero in on design solutions. On top of all this, with agile testing there is rarely time to build the giant spreadsheets of test cases that management loves to use for tracking testing progress. So, how do we handle the constant flood of change, balance exploratory testing and automated testing—and communicate progress to senior management? Matt Attaway shares his experiences developing a successful lightweight agile testing process that helped his team move from testing two waterfall projects at once to six agile projects simultaneously. Matt discusses lightweight test plan development, the use of BDD tools such as Cucumber, and methods for measuring testing progress.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Matthew Attaway, Perforce Software

  1. 1.       nt Session    Presented by:  Matthew       Brought to you by:      340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 T11  Concurre 4/8/2014    2:00 PM          “Balancing Exploratory and   Automated Testing in Agile”       Attaway Perforce Software              300, 68‐8770 ∙ 904‐278‐0524 ∙ ∙ 
  2. 2. Matthew Attaway Perforce Software   A jack of all trades and master of some, Matt Attaway has worked as a tester, developer, researcher, designer, manager, DevOps engineer, and elephant trainer. He currently manages the open source development group at Perforce Software, but prior to that he led a test team working on the highest profile projects at Perforce. When not tinkering with the latest software development pipeline tools, Matt enjoys experimenting with cocktails and playing games of all sorts—preferably simultaneously.
  3. 3. 2/18/2014 1 Balancing Exploratory and Automated Testing in Agile Matt Attaway | @mataway
  4. 4. 2/18/2014 2 • Projects varied in size from 3-20 people • Multiple languages, multiple stacks My World • GUIs, CLI tools, servers, and APIs • Remote and co-located teams • Scrum, Kanban, Scrumban, Waterscrumfallban • Managed ~10 testers on 6 simultaneous projects • Almost all "bet the company" projects • Automation or bust! My Constraints • Exploratory testing a must • Volatile projects; few design docs • Extremely high bar for quality • Executive demand for measurability and metrics • Do more with less
  5. 5. 2/18/2014 3 Must handle the constant flood of change My Process Requirements • Must handle the constant flood of change • Must integrate automated testing without sacficing exploratory testing • Must communicate progress both to the product team and the executive teamteam and the executive team Must not die
  6. 6. 2/18/2014 4 Ruthlessly optimize Exploratory testing T t l Automated testing Test plans
  7. 7. 2/18/2014 5 Exploratory testing T t l Automated testing Test plans Helm th on Moltke the Elder “No plan survives contact with the enemy.” –Helmuth von Moltke the Elder
  8. 8. 2/18/2014 6 Needed test plans per epic story feature and fix• Needed test plans per epic, story, feature and fix • Easy report generation and APIs • Easy access for testers Cheap both in time and in money• Cheap, both in time and in money Issue tracker
  9. 9. 2/18/2014 7 Focused around epic story feature and fix Strengths • Focused around epic, story, feature and fix • Built-in report generation and APIs • Testers already lived in it Already installed and paid for!• Already installed and paid for! Test plans highly visible to the developers Bonuses • Test plans highly visible to the developers • Naturally supports multiple editors • Extremely easy to extend • Notifications of updates via email and RSS with diffs• Notifications of updates via email and RSS, with diffs
  10. 10. 2/18/2014 8 • Added a "Testing Notes" field to issues Testers added test cases as they did their Process • Testers added test cases as they did their exploratory testing • + used to flag completed test, - to flag pending test • Anyone could add test cases to an issue • Tweet sized test cases; should be clear for knowledgable tester Example - Attempt to create user with HTTP special characters - Attempt to create user with XML special characters +Attempt to create user that already exists Example
  11. 11. 2/18/2014 9 • Devs handed off work by merging to mainline Process & Measuring • Issue not closed until all test cases complete • Outstanding test work = issues in 'fixed' state • Test work displayed on burndown charts; easy to spot when test was having a hard time keeping up
  12. 12. 2/18/2014 10 Automated testing • Numerous automated test technologies and languages CLI t t t l b t t t l d GUI t t t l• CLI test tools, web test tools, and GUI test tools • Poorly written code; copy/paste gone wild • Each product had one (or more!) test frameworks • Too slow to add tests• Too slow to add tests • Tests were dependent on each other
  13. 13. 2/18/2014 11
  14. 14. 2/18/2014 12 Given I am speaking at StarCanada When I start my talk Then my heart rate will go up dramatically • Forced code modularization Why Cucumber? • Test cases are legible and shareable • Can drive any tool; can use most languages • Tests are easy to add; frequently require no "real" code • Works out of the box; huge development community W it t t ti• Write tests anytime • Everyone can automate • Cucumber is, dare I say, fun
  15. 15. 2/18/2014 13 • Hour meeting at the beginning of the sprint; more as Bugstorming needed during sprint • Wrote Gherkin as a team coming up with different attacks based on slotted stories • Helped to share system knowledge and automated testing knowledge • Track your hit rate; failed automated test is a bug T k h t t t d h t d 't ( d h !) Automated testing advice • Track what you automate and what you don't (and why!) • Aim to be able to add a test case in 15 minutes • Always Reset State • Automate all the things• Automate all the things • Plan for parallelized tests from day one
  16. 16. 2/18/2014 14 Ruthlessly optimize Avoid documents that do nothing (or make your documents do more)
  17. 17. 2/18/2014 15 Embrace your issue tracker Detach writing automated tests from implementing automated tests
  18. 18. 2/18/2014 16 Regression tests == automated tests The Cucumber Book by Aslak Hellesøy Books • The Cucumber Book by Aslak Hellesøy • Cucumber & Cheese by Jeff Morgan • Continuous Delivery by Jez Humble & David Farley Exploratory Software Testing by James Whittaker• Exploratory Software Testing by James Whittaker