Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Agile Testing Dilemmas

1,784 views

Published on

Based on hands-on Agile experience acquired over multiple delivery projects and Agile coaching/consulting assignments, Vatsala and Aman share common Agile Testing dilemmas, and possible solutions, tying them to the principle of "moving testing upstream".

Presented at Next Generation Testing conference in Bangalore, India (July 2014).

Published in: Technology
  • Be the first to comment

Agile Testing Dilemmas

  1. 1. Breaking Free of Agile Testing Dilemmas! Vatsala Singh Aman King
  2. 2. What’s a dilemma?
  3. 3. di·lem·ma noun də-ˈle-mə a situation in which you have to make a difficult choice http://www.m-w.com/dictionary/dilemma
  4. 4. What makes Agile Testing a dilemma?
  5. 5. Few Reasons “Continuous delivery of valuable software” “Welcome changing requirements” “Face-to-face conversation” “Working software is the primary measure of progress”
  6. 6. Agile Manifesto February 2001, Utah, USA http://agilemanifesto.org/principles.html
  7. 7. Lean Muda (無駄) 7 wastes “Whenever defects occur, extra costs are incurred… should not be passed on to the consumer and should be taken as a loss.” 2001 2003 http://en.wikipedia.org/wiki/The_Toyota_Way http://en.wikipedia.org/wiki/Lean_software_development
  8. 8. “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” W. Edwards Deming (1900 – 1993) https://www.deming.org/theman/theories/fourteenpoints
  9. 9. Cost of Change: In requirement phase x1 in analysis x10 in design x100 and so on 1981 http://www.academia.edu/1554649/Software_Engineering_Economics_by_Barry_W._Boehm
  10. 10. Dilemma! In essence… testing a developed product is already too late?!
  11. 11. 0 20 40 60 80 100 120 Requirements Design & Implementation Testing Cost of Defect Testing Upstream Move testing upstream!
  12. 12. Analyst Business Developer QA Requirements Testing Design & Implementation Testing $$$$$$ $$$$$$
  13. 13. Testing Upstream Testing Testing Analyst Business Developer QA Requirements Testing Testing Design & Implementation $$$ $$$$$$ $$$$$$ $$$$$$
  14. 14. Iterations Analyst Business Developer QA Requirements Design & Implementation Testing Testing Testing Testing $ $$ $$$$
  15. 15. Iterations Analyst Business Developer QA Requirements Design & Implementation Testing Testing Testing Testing How? How? How?
  16. 16. Business Facing Technology Facing
  17. 17. Business Facing Technology Facing SupportProgramming CritiqueProduct
  18. 18. Agile Testing Quadrants http://www.exampler.com/old-blog/2003/08/21/#agile-testing-project-1 Brian Marick
  19. 19. Quadrant Dilemmas •Automation? •Manual? What •Requirements? •Design & Implementation? •Testing? When •Business Analysts? •Developers? •QAs? Who
  20. 20. Agile Testing Quadrants Business Facing Technology Facing SupportProgramming CritiqueProduct Desk-check Showcase / UAT Exploratory Testing Usability Testing Performance Tests Security Tests Unit Tests Component Tests Integration Tests Paper Prototyping / UX Functional UI Tests Acceptance Tests Automated + Manual Manual Automated Automated + Manual What
  21. 21. Agile Testing Quadrants Business Facing Technology Facing SupportProgramming CritiqueProduct Design & Implementation Requirements Design & Implementation Design & Implementation Testing Post-Deployment Design & Implementation Testing Post-Deployment When
  22. 22. Agile Testing Quadrants Business Facing Technology Facing SupportProgramming CritiqueProduct Business Analysts QAs Developers QAs End-users Business Analysts QAs Developers QAs Who
  23. 23. http://agiletester.ca/ 2009
  24. 24. End to end – business facing Localized – technology facing Test Pyramid Unit Service UI http://martinfowler.com/bliki/TestPyramid.html
  25. 25. Test Pyramid Broaderscope Fasterfeedback Unit Service UI
  26. 26. Anti-pattern: Ice-cream Cone Unit Service UI
  27. 27. A closer look at common dilemmas
  28. 28. PracticePrinciple Moving testing upstream TDD ATDD Effective manual testing Timely feedback
  29. 29. Practice SolutionProblem ATDD
  30. 30. Practice SolutionProblem ATDD Needs close collaboration Slow tests Test maintenance Test data Flaky tests
  31. 31. Practice SolutionProblem ATDD Slow tests Test maintenance Test data Flaky tests Story kickoff Design tests before development Definition of “Done” No “stabilization” phase Needs close collaboration
  32. 32. Practice SolutionProblem ATDD Test maintenance Test data Flaky tests Needs close collaboration Test pyramid Parallelization Headless browser testing Slow tests
  33. 33. Practice SolutionProblem ATDD Slow tests Test data Flaky tests Needs close collaboration Test maintenance Test pyramid Tests consolidation Test reviewers/ champions
  34. 34. Practice SolutionProblem ATDD Slow tests Test maintenance Flaky tests Needs close collaboration Test data Subset of prod data Setup/ teardown scripts
  35. 35. Practice SolutionProblem ATDD Slow tests Test maintenance Test data Needs close collaboration Flaky tests Conditional waits Test data availability Dependent systems check Stubbed dependencies
  36. 36. Practice SolutionProblem Efficient manual testing
  37. 37. Practice SolutionProblem Efficient manual testing Balance automation vs manual Too many manual scenarios Lack of domain knowledge
  38. 38. Practice SolutionProblem Efficient manual testing Balance automation vs manual Too many manual scenarios Lack of domain knowledge Devs collaborate on automation Manual test what cannot be automated Automate in parallel to development Dev : QA ratio
  39. 39. Practice SolutionProblem Efficient manual testing Balance automation vs manual Too many manual scenarios Lack of domain knowledge Scope to current iteration Consolidate manual scenarios More automation
  40. 40. Practice SolutionProblem Efficient manual testing Balance automation vs manual Too many manual scenarios Lack of domain knowledge Pair with Business Analyst Run Client Showcases Training
  41. 41. Practice SolutionProblem Timely feedback
  42. 42. Practice SolutionProblem Missing CI practices Lack of CI environments “Scheduled” CI builds Timely feedback Delayed testing
  43. 43. Practice SolutionProblem Lack of CI environments “Scheduled” CI builds Timely feedback Delayed testing CI tool setup Training Automate in parallel to development Missing CI practices
  44. 44. Practice SolutionProblem “Scheduled” CI builds Timely feedback Delayed testing Provision dedicated VMs Infrastructure as Code Database schema on demand Missing CI practices Lack of CI environments
  45. 45. Practice SolutionProblem Timely feedback Delayed testing Trigger build per commit Pull out slow stages into parallel builds Composite builds Missing CI practices Lack of CI environments “Scheduled” CI builds
  46. 46. Practice SolutionProblem Timely feedback Manual desk-check testing CI deployment pipeline No “stabilization” phase Missing CI practices Lack of CI environments “Scheduled” CI builds Delayed testing
  47. 47. But how do we know we’re making progress?
  48. 48. Metrics Customer-reported defect count Average time between faulty commit and defect detection Runtime of tests Lead time of the entire process
  49. 49. What’s bothering you today? Thank you slideshare.net/amanking thoughtworks.com

×