The document outlines a framework for automation testing presented by KMS Technology, detailing challenges, solutions, and advanced features. It highlights the importance of metrics and ROI in automation, emphasizing cost reduction, improved product quality, and bug escape reduction. The framework leverages flexible scripting tools and multi-platform support, enabling efficient and effective test automation for various applications.

1. FRAMEWORK FOR AUTOMATION
TESTING – PRACTICE SHARING
June 2013
KMS Technology: http://kms-technology.com

2. AGENDA
Duong Phan – QA Architect
 Introduction
 Challenges and Solutions
 KMS Test Automation Framework
 Advanced Features
Vu Duong – Engineering Manager
 ROI and Metrics
2

3. KMS TECHNOLOGY OVERVIEW
3
US Company/Passionate Global Workforce
• 400 Resources & Growing in Vietnam and the US
• 160 Testers ~ 50% Workforce
• Proven Leadership Team
World Class Infrastructure
• Built for ISO 27001, Planned Certification in 2013
Best-in-Class SDLC Practices
• CMMI and Agile focus
• QASymphony - Commercial Agile Testing Solutions
recognized by Forrester with over 4,500 users
Best Clients – Raving Fans
• 100% Referenceable and Ecstatic
• 100% in long-term dedicated team

4. A CASE STUDY
4
About the Client
is a leader in eCommerce software and
solutions, Provide multifaceted
eCommerce platform, reliable 24/7/365
service, scalable IT infrastructure,
integration with third party, and custom-
built features, Serve over 160 retail sites
Technology:
 Platform: Windows XP, Windows 7,
Linux, OSX, iOS and Android.
 Technology: Java, .NET/C#, Oracle
 Testing Tools: JIRA, Testlink, KMS
Automation Framework
Achievements
 KMS is responsible for automation of
client Core Platform Product and also for
distribution platforms
 Reusable Automation Framework can
stand independent with KMS run-time
plug-in (support web testing) although
can adapt other tools (Selenium)
 Framework support multiple OS:
Windows, Linux and OSX, iOS and
Android
 Execute test daily, Automation script is
able to find a lot of defects
Confidential
Automation Goals
 Reduce cost of development and quality
assurance
 Continuous platform upgrade to provide
new features and patch releases
 Testing is performed on core platform
and more than 200 of distributions
 Testing on many browsers (IE, Firefox,
Chrome, Safari)

5. CHALLENGES AND OUTCOMES
 Project Specification
 Multiple Instance Application Types
 Flexible Components
 Regular Sanity Test
 Automation must find bugs
 Customer Satisfaction
 Cost Saving
 Product Quality Improvement
 Bug Escape Reduction
5

6. KEY FEATURES…
Framework using Excel as Scripting tool
• Easy to use and learn
• Easy to apply Data Driven and Keyword Driven ideas
Prototype plug-in
• Able to develop wrapper plug-in with several tools
• One script can run under any plug-ins
Log System
• General report and log types
• History report
6

7. KMS TEST AUTOMATION FRAMEWORK
7

8. EXECUTION MODEL AND PLUG-INS
8

9. SCRIPTING
9

10. © 2013 KMS Technology
ADVANCED FEATURES

11. SMART OBJECT RECOGNITION
Multiple GUI instances
 1 GUI for many kinds of instance objects, apply for multi
sites product
 Example: 1 gui object txt_Email may be detect for 4 sites (gmail,
yahoo mail, facebook, Linkedin)
11

12. KEYWORD DRIVEN - FLEXIBLE REUSABILITY
12
 A library can use other libraries
 Test case can use any library
 Test case can call other test cases
 Data-driven on test case calls test cases

13. DATA DRIVEN - FLEXIBLE CONFIGURATION
13
 Multiple Datasets

14. DATA DRIVEN - FLEXIBLE CONFIGURATION
14
 Able to switch runtime engines
 Sample case
• Configure steps in IE
• Execute some steps in Firefox
• Verify result steps on Chrome

15. LOG AND REPORT
15

16. LOG AND REPORT
History report (excel, html and email)
 Helpful to follow bug fixing
 HTML format can be opened anywhere
 Email (HTML format) is portable
Various type of reports/logs
 Statistic of result
 Detail report/log
 HTML / Excel format report
 Reminders
 Send email notification
 Define groups to send notification
16

17. LOG AND REPORT
 Follow Drill-down idea
History Report  Step report  Detail report
17

18. © 2013 KMS Technology
ROI & METRICS
VU DUONG

19. ROI – WHAT?
19

20. ROI – WHAT?
20

21. ROI – WHY?
21

22. ROI – WHY?
22

23. ROI – HOW?
23

24. ROI ANALYSIS OVERVIEW
24
To calculate ROI, the benefit (return) of an
investment is divided by the cost of the
investment; the result is expressed as a
Percentage or a Ratio.

25. ROI – BEST PRACTICE
25
 ROI (Effort)
 ROI (Cost)

26. ROI - EFFORT
26
ESV (Effort saved from using
automation)
ROI (effort) = ------- = (-------------------------------------------------
)
ESP (Effort involved in automation)
 ESV = Effort saved as a result of test execution using automation (in
hours)
 ESP = Effort involved in automation like: Framework Development,
Automation Scripts Development, Automated Script Execution,
Automation Scripts Maintenance, Automation Training …

27. ROI ANALYSIS – EFFORT
27
 Parameters:

28. ROI ANALYSIS – EFFORT
28
Pay-Back Period (Effort)

29. ROI ANALYSIS – COST
29
Parameters:

30. ROI ANALYSIS – COST
30
Pay-Back Period (Cost)

31. METRICS
31

32. METRICS – EFFORT/COST
32
 Metrics for Effort/Cost
 Running Time Per Month
 Critical/High Defect Per Component
 Critical/High Defect Per Month

33. METRICS – QUALITY
33
 Metrics for Quality
 Defect Escape
 % Automation Coverage Per Sites/Features
 Defect Found by Automation
 Defect Rate (Compare with Manual)
 …….

34. © 2013 KMS Technology
THANK YOU
34