Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Framework For Automation Testing Practice Sharing

1,405 views

Published on

This is the presentation that Mr. Duong Phan and Mr. Vu Duong shared at KMS's public seminar on Saturday, June 15, 2013.

Published in: Technology, Business
  • Be the first to comment

Framework For Automation Testing Practice Sharing

  1. 1. FRAMEWORK FOR AUTOMATION TESTING – PRACTICE SHARING June 2013 KMS Technology: http://kms-technology.com
  2. 2. AGENDA Duong Phan – QA Architect  Introduction  Challenges and Solutions  KMS Test Automation Framework  Advanced Features Vu Duong – Engineering Manager  ROI and Metrics 2
  3. 3. KMS TECHNOLOGY OVERVIEW 3 US Company/Passionate Global Workforce • 400 Resources & Growing in Vietnam and the US • 160 Testers ~ 50% Workforce • Proven Leadership Team World Class Infrastructure • Built for ISO 27001, Planned Certification in 2013 Best-in-Class SDLC Practices • CMMI and Agile focus • QASymphony - Commercial Agile Testing Solutions recognized by Forrester with over 4,500 users Best Clients – Raving Fans • 100% Referenceable and Ecstatic • 100% in long-term dedicated team
  4. 4. A CASE STUDY 4 About the Client is a leader in eCommerce software and solutions, Provide multifaceted eCommerce platform, reliable 24/7/365 service, scalable IT infrastructure, integration with third party, and custom- built features, Serve over 160 retail sites Technology:  Platform: Windows XP, Windows 7, Linux, OSX, iOS and Android.  Technology: Java, .NET/C#, Oracle  Testing Tools: JIRA, Testlink, KMS Automation Framework Achievements  KMS is responsible for automation of client Core Platform Product and also for distribution platforms  Reusable Automation Framework can stand independent with KMS run-time plug-in (support web testing) although can adapt other tools (Selenium)  Framework support multiple OS: Windows, Linux and OSX, iOS and Android  Execute test daily, Automation script is able to find a lot of defects Confidential Automation Goals  Reduce cost of development and quality assurance  Continuous platform upgrade to provide new features and patch releases  Testing is performed on core platform and more than 200 of distributions  Testing on many browsers (IE, Firefox, Chrome, Safari)
  5. 5. CHALLENGES AND OUTCOMES  Project Specification  Multiple Instance Application Types  Flexible Components  Regular Sanity Test  Automation must find bugs  Customer Satisfaction  Cost Saving  Product Quality Improvement  Bug Escape Reduction 5
  6. 6. KEY FEATURES… Framework using Excel as Scripting tool • Easy to use and learn • Easy to apply Data Driven and Keyword Driven ideas Prototype plug-in • Able to develop wrapper plug-in with several tools • One script can run under any plug-ins Log System • General report and log types • History report 6
  7. 7. KMS TEST AUTOMATION FRAMEWORK 7
  8. 8. EXECUTION MODEL AND PLUG-INS 8
  9. 9. SCRIPTING 9
  10. 10. © 2013 KMS Technology ADVANCED FEATURES
  11. 11. SMART OBJECT RECOGNITION Multiple GUI instances  1 GUI for many kinds of instance objects, apply for multi sites product  Example: 1 gui object txt_Email may be detect for 4 sites (gmail, yahoo mail, facebook, Linkedin) 11
  12. 12. KEYWORD DRIVEN - FLEXIBLE REUSABILITY 12  A library can use other libraries  Test case can use any library  Test case can call other test cases  Data-driven on test case calls test cases
  13. 13. DATA DRIVEN - FLEXIBLE CONFIGURATION 13  Multiple Datasets
  14. 14. DATA DRIVEN - FLEXIBLE CONFIGURATION 14  Able to switch runtime engines  Sample case • Configure steps in IE • Execute some steps in Firefox • Verify result steps on Chrome
  15. 15. LOG AND REPORT 15
  16. 16. LOG AND REPORT History report (excel, html and email)  Helpful to follow bug fixing  HTML format can be opened anywhere  Email (HTML format) is portable Various type of reports/logs  Statistic of result  Detail report/log  HTML / Excel format report  Reminders  Send email notification  Define groups to send notification 16
  17. 17. LOG AND REPORT  Follow Drill-down idea History Report  Step report  Detail report 17
  18. 18. © 2013 KMS Technology ROI & METRICS VU DUONG
  19. 19. ROI – WHAT? 19
  20. 20. ROI – WHAT? 20
  21. 21. ROI – WHY? 21
  22. 22. ROI – WHY? 22
  23. 23. ROI – HOW? 23
  24. 24. ROI ANALYSIS OVERVIEW 24 To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a Percentage or a Ratio.
  25. 25. ROI – BEST PRACTICE 25  ROI (Effort)  ROI (Cost)
  26. 26. ROI - EFFORT 26 ESV (Effort saved from using automation) ROI (effort) = ------- = (------------------------------------------------- ) ESP (Effort involved in automation)  ESV = Effort saved as a result of test execution using automation (in hours)  ESP = Effort involved in automation like: Framework Development, Automation Scripts Development, Automated Script Execution, Automation Scripts Maintenance, Automation Training …
  27. 27. ROI ANALYSIS – EFFORT 27  Parameters:
  28. 28. ROI ANALYSIS – EFFORT 28 Pay-Back Period (Effort)
  29. 29. ROI ANALYSIS – COST 29 Parameters:
  30. 30. ROI ANALYSIS – COST 30 Pay-Back Period (Cost)
  31. 31. METRICS 31
  32. 32. METRICS – EFFORT/COST 32  Metrics for Effort/Cost  Running Time Per Month  Critical/High Defect Per Component  Critical/High Defect Per Month
  33. 33. METRICS – QUALITY 33  Metrics for Quality  Defect Escape  % Automation Coverage Per Sites/Features  Defect Found by Automation  Defect Rate (Compare with Manual)  …….
  34. 34. © 2013 KMS Technology THANK YOU 34

×