Building Test Automations the agile way


Published on

An alternate approach towards building Test Automations for the Agile development environments. Traditional Test Automation approach isn't a great fit in fast pace dynamic ecosystem, hence there exists a dire need for an alternate approach.

This is very well suited to an e-commerce platform, and can be easily extended to similar environments.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building Test Automations the agile way

  1. 1. Bringing Agility in Test Automations Hemant Kumar Jaggi Senior Manager,
  2. 2. LET‟S BE AGILE ! Heard this before?
  3. 3. Thoughts on test automations the “Agile” way To keep up the pace with agile application development Traditional Test Automation approach isn‟t working Best practices must be incorporated everywhere Ideally suited to an Ecommerce (or similar) environment
  4. 4. An Ecommerce looks like
  5. 5. Only thing that doesn’t change is “change”
  6. 6. Building Test Automations the “Agile” way
  7. 7. WHY?
  9. 9. WHY? Challenges we face @ MakeMyTrip Fast Pace Releases in a week? Monthly unique visitors? Monthly page views? High Maintenance Cost & Test cases not always in “shape”
  10. 10. Challenges (contd.) Complex Business Rules Hotel promotion combinations? Cheapest Flights search? Multiple GDS connections Hotel Search engine? ~10,500 Domestic Hotels ~250,000 hotels across world It‟s impossible to reverse-engineer!
  11. 11. Challenges (contd.) Execution issues: Automation execution may take 2-3 days to complete Golden data set takes ~7 days to build Automation teams don‟t know domain Finally ..
  12. 12. Life of a QA Engineer is like
  13. 13. WHAT?
  14. 14. WHAT it should be? Faster time to develop Automation LOW maintenance cost Agnostic to business logic “QA team” friendly Lesser complexity Faster execution time “It just works ! ”
  15. 15. HOW?
  16. 16. HOW? In lines with Agile Manifesto.. Break into smaller stories Prioritize them basis Functional Importance Frequency of executions Dependencies Work “with” QA team Develop a story and deliver
  17. 17. Create Buckets Sanity tests End to end user flows Regression tests* Miscellaneous flows
  18. 18. Sanity Test Breadth wise test coverage. MATS – Minimal Acceptance Test Suite Release checklist Determines whether build is stable.
  19. 19. User Flows Covers end to end user flows Detailed breadth wise coverage E.g. different combinations of booking flows.
  20. 20. But how to simplify regression tests? Break them further .. Into smaller flows, e.g. page wise test suites Comparative Testing Interactive Testing
  21. 21. Comparative Testing
  22. 22. Comparative Testing Works on two builds: “QA” build (i.e. build under test) “Production” build Deploy them in same environment. Read all „to-be-tested‟ values from front end
  23. 23. Comparative Testing Execute automation script on both builds Dump them into XML (or excel) Excel template has formulae to compare values from two builds Or use an XML comparator (in case of XML output format)
  24. 24. Comparative Testing IF (Actual == Expected) THEN PASS ELSE FAIL * Actual Value = value(QA build) * Expected Value = value(live build) Take XML(or excel) as input to reporting framework, generate HTML reports
  25. 25. Let’s discuss Pros & Cons of Comparative Testing
  26. 26. Advantages of Comparative Testing Faster turn around to develop Automation script does just what it is supposed to do Automation becomes a “dumb” execution Very LOW maintenance cost Keep it simple
  27. 27. Advantages of Comparative Testing Agnostic to business layers Changes in business rules don‟t matter High level of accuracy Makes it “thorough” Enhances confidence in QA team
  28. 28. Advantages of Comparative Testing Overall progress is faster More dependency on Test Scenarios than Test Cases Less/No domain knowledge required for Automation team “Win-Win” for Automation and QA team
  29. 29. Disadvantages of Comparative Testing Any “live” issues won‟t be caught Execution requires Either double the time (if sequential) Or double the resources (if parallel)
  30. 30. Interactive Testing
  31. 31. Interactive Testing Complements comparative testing Features that require user interaction E.g. Filters, Sorting etc. Comparative + Interactive equals “Regression”
  32. 32. Automation Coverage Tracking Assign weights to each bucket, say Automation Score = sum([(Tn/An)*100]*[Wn/(W1+W2+W3+W4+W5)] Tn=Total Automated TC’s for a bucket An=Total Automation feasible TC’s for a bucket Wn=Weighted score of a bucket
  33. 33. Automation Coverage Tracking (Sample)
  34. 34. Computing efficiency gains (Sample)
  35. 35. Take Away The “Agile” way of building Test Automations We resolved “challenges” of traditional automation approach Simplifying regression Regression=Comparative + Interactive Its working @ MakeMyTrip
  36. 36. Thank You Hemant Jaggi