Application & Script Independent Framework: The Need For Data Normalization ASIF Date: Wednesday, 14 th November, 2007 Venue: Orlando, Florida Presenter: Raj Kamal Company: Microsoft, India

Name: Raj Indian citizen 25 years First time in US Flew over 7718 + miles to attend QAI Basically from a village in Haryana , India. Sports – Ping Pong, Cricket and Pool Like learning new technologies and gadgets Who am I ? Born in Delhi Work @ MS Hyderabad

Name: Raj

Indian citizen

25 years

First time in US

Flew over 7718 + miles to attend QAI

Basically from a village in Haryana , India.

Sports – Ping Pong, Cricket and Pool

Like learning new technologies and gadgets

Research

Past Research Iterative Automation Process using RUP ART (Automation Review Tool ) Mapping RUP to Rational Testing Tools Presented at http://ifstep.org/ ROI of Testing http://geektester.blogspot.com

Iterative Automation Process using RUP

ART (Automation Review Tool )

Mapping RUP to Rational Testing Tools

Presented at http://ifstep.org/

ROI of Testing

Big Question

Agenda Industry Quotes Problems Solution Architecture Implementation Benefits

Industry Quotes

Problems

Solution

Architecture

Implementation

Benefits

Survey “ Requirement Changes are inevitable”

“ In QAI survey w hen asked, “ To what degree are the test products maintained and reused ?” 33% said always, 59% sometimes, and 8% never ” ” Only 19.4% of respondents said they used some type of automated testing tool, the remainder relying upon manual testing methods.” “ The prevailing reasons given for not automating test processes were: Time (35%); cost (33%); script/data maintenance (26%); and complexity (18%) “ Survey: Big Spends, Little Comfort

“ requirements; the real joke is that the requirements end up being what the source code does.” “ 62 % of respondents stated they could be driving a new Mercedes  or retired if they had a dollar ($) for every time they had to rewrite a test case because requirements changes weren't effectively communicated "it is often the case that development can slip on their timelines, but testing cannot.“ Survey: Automation is everywhere

Available Automation Frameworks

What’s Available Record / Playback Data Driven Framework Modular Architecture Keyword Driven Framework Hybrid Framework http://en.wikipedia.org/wiki/Test_Automation_Framework

Record / Playback

Data Driven Framework

Modular Architecture

Keyword Driven Framework

Hybrid Framework

http://en.wikipedia.org/wiki/Test_Automation_Framework

Low ROI & High Maintenance Effort ITERATIVE DEVELOPMENT REDUNDANCY INCONSISTENCY PROBLEM: CAUSES OF AUTOMATION FAILURE Frequent updates Multiple versions Difficult to Debug Common Test Data Automation Users Complexity Grows Change Requests Phases Implementation Frequent Script updates SKILLS People Issues Common Flow Frequent Failures Coding

Problem #1: Redundancy Scenario: Test Data Sets : 500 Tables / Sheets Test Cases Covered: 1000 Testers: 4 Modules: 4 Unintentional Repetition of Test Data & Multiple copies of the same data maintained

Scenario:

Test Data Sets : 500 Tables / Sheets

Test Cases Covered: 1000

Testers: 4

Modules: 4

Problem #1: Redundancy Future Release Changes - Object Type - Object Name - Object Data Impact / Loss [ Domino effect ] - Multiple Script Failure - Rework updating multiple instances - Loss of Data Integrity

Future Release Changes

- Object Type

- Object Name

- Object Data

Impact / Loss [ Domino effect ]

- Multiple Script Failure

- Rework updating multiple instances

- Loss of Data Integrity

Problem 2#: Inconsistency Future Release Changes - Object Type - Object Name - Object Data Impact / Loss Inconsistency – Different versions of Data Tedious Process Misleading Result / Invalid Bugs

Future Release Changes

- Object Type

- Object Name

- Object Data

Impact / Loss

Inconsistency – Different versions of Data

Tedious Process

Misleading Result / Invalid Bugs

Problem #3: Evolving Development Models Agile Modeling SCRUM XP RUP Iterative Development Microsoft Solutions Framework And Many More …..

Agile Modeling

SCRUM

XP

RUP

Iterative Development

Microsoft Solutions Framework

And Many More …..

Problem #3: Evolving Development Models Evolutionary Dev Models can handle changes gracefully and are becoming better every day…. But Is our automation evolving / adapting with the same pace ?

Evolutionary Dev Models can handle changes gracefully and are becoming better every day….

Solution

Suggested Solution : ASIF Data Normalization User defined Views Other RDBMS concepts 100 % Reusable Plug and Play Library Simplified Scripts (just to invoke library) “ ASIF, just like Rome, wasn't built in a day”

Data Normalization

User defined Views

Other RDBMS concepts

100 % Reusable Plug and Play Library

Simplified Scripts (just to invoke library)

Solution #1: Normalization Minimize duplication by storing only ONE copy of the Test Data Object Type Object Name Object Data Have Test Data in Normal Form – 1 NF, 2 NF, 3 NF Eliminate Redundancy, Inconsistency

Minimize duplication by storing only ONE copy of the Test Data

Object Type

Object Name

Object Data

Solution #2: User Defined Views 1 View ~ 1 Test Case View = Table 1 + Table 2 + .. (JOIN) View contain “FLOW” of Test Execution Table Updated ~ Views get updated Views get Updates ~ Tables get updated Easy to create / drop / modify Contains BUSINESS LOGIC

1 View ~ 1 Test Case

View = Table 1 + Table 2 + .. (JOIN)

View contain “FLOW” of Test Execution

Table Updated ~ Views get updated

Views get Updates ~ Tables get updated

Easy to create / drop / modify

Contains BUSINESS LOGIC

Solution #3: Reusable Library ASIF Library Talks with Views NOT Tables Can Read / Understand / Update Views Perform ACTION / VERIFICATION Handles Errors & Exceptions Can write logs / results (PASS / FAIL) 100 % Reusable – DON’T CONTAIN BUSINESS LOGIC Interface between Scripts & Test Data

ASIF Library

Talks with Views NOT Tables

Can Read / Understand / Update Views

Perform ACTION / VERIFICATION

Handles Errors & Exceptions

Can write logs / results (PASS / FAIL)

100 % Reusable – DON’T CONTAIN BUSINESS LOGIC

Interface between Scripts & Test Data

Solution #4: Over Simplified Scripts Scripts Invoke ASIF Reusable Library Passes “Test Case Id” as Parameter DON’T CONTAIN BUSINESS LOGIC DON’T CONTAIN ACTION CODE DON’T CONTAIN VERIFICATION CODE 90 % Reduction in LOC (Lines of Code)

Scripts

Invoke ASIF Reusable Library

Passes “Test Case Id” as Parameter

DON’T CONTAIN BUSINESS LOGIC

DON’T CONTAIN ACTION CODE

DON’T CONTAIN VERIFICATION CODE

90 % Reduction in LOC (Lines of Code)

Implementation

ASIF Automation Architecture Name Age Views Tables Scripts

Implementation: Automation Process Setup

Implementation: Database Setup

Implementation: Automation Test Tool

General Flow PK Object Name / Type Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox UName~EditBox TC_01 User1 Pwd1 Raj TC_02 User2 Pwd2 Sam

Database Design User Id Password XYZ Login Page OK Cancel User name : Age : XYZ User Information Raj 25 PK Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox TC_01 User1 Pwd1 TC_02 User2 Pwd2 PK Object Name / Type Object Name / Type 0R UName~EditBox UAge~EditBox TC_01 Raj 25 TC_02 Sam 28 PK Object Name / Type Object Name / Type 0R Userid~EditBox UName~EditBox TC_01 User1 Raj TC_02 User2 Sam

Example #1 – Change in Requirements Requirement changes the business flow or logic Updated only views written for that test case Tables get updated automatically Scripts need not to be touched. Run scripts and validate the changes

Requirement changes the business flow or logic

Updated only views written for that test case

Tables get updated automatically

Scripts need not to be touched.

Run scripts and validate the changes

Example #2 – Change in Design/Code Object name / Object type is added / changed by a developer Updated base table only at one place Views get updated automatically Scripts need not to be touched. Run scripts and validate the changes

Object name / Object type is added / changed by a developer

Updated base table only at one place

Views get updated automatically

Scripts need not to be touched.

Run scripts and validate the changes

Cost

Cost – Manual Testing Cost of manual Testing = time to develop test cases + time to maintain test cases x number of times tests are executed + time to execute manual testing x number of times tests are executed

Cost of manual Testing

= time to develop test cases

+ time to maintain test cases x number of times tests are executed

+ time to execute manual testing x number of times tests are executed

Cost- Automation using ASIF Cost of automation Testing = price of hardware + price of software + time to develop scripts + time to maintain scripts x number of times scripts are executed + time to execute scripts x number of times scripts are executed

Cost of automation Testing

= price of hardware

+ price of software

+ time to develop scripts

+ time to maintain scripts x number of times scripts are executed

+ time to execute scripts x number of times scripts are executed

Return on Investment

ASIF Return on Investment ROI = (cost of manual – cost of automation) / cost of automation ASIF gives ROI of 72 % over the course of the year

ROI = (cost of manual – cost of automation) / cost of automation

Benefits

Benefits Easy maintainability Test Data Storage and Retrieval Efficiency Better test coverage Modularization Gracefully handles scalability

Easy maintainability

Test Data Storage and Retrieval Efficiency

Better test coverage

Modularization

Gracefully handles scalability

ASIF Break-even Point Test Runs Cost of Testing n En = (Va + n * Da )/ (Vm + n * Dm ) V - Expenditure for Implementation D - Expenditure for single test execution

Benefits Accelerated time to Market Abstract and decouple test data from scripts Provides 100 % Reusable Plug & Play Library Reduced cost of quality Decouple complex business function

Accelerated time to Market

Abstract and decouple test data from scripts

Provides 100 % Reusable Plug & Play Library

Reduced cost of quality

Decouple complex business function

Effort Reduction Releases Effort Application Complexity - Test Effort Reduction over Releases

Overall Performance Total Effort (%) Complexity Total Effort Reduction (%)

Why ASIF and not any other automation framework ? Almost 90 % reduction in the script code Handles requirement changes gracefully ASIF Automation framework don’t take breaks, don’t need to eat, don’t take calls from their family 

Almost 90 % reduction in the script code

Handles requirement changes gracefully

ASIF Automation framework don’t take breaks, don’t need to eat, don’t take calls from their family 

Conclusion: When to Use ? Requirements are Dynamic Ideal for Iterative / Agile / SCRUM Model At least 3 Test Runs to obtain ROI Testers lack Coding skills Ideal to run Regression and BVT Suite Permanent long term solution for Client

Requirements are Dynamic

Ideal for Iterative / Agile / SCRUM Model

At least 3 Test Runs to obtain ROI

Testers lack Coding skills

Ideal to run Regression and BVT Suite

Permanent long term solution for Client

Future Avenues of Research

Think New Leverage artificial intelligence Simplify ( Reduce Complexity ) - Adoption - Use - Maintenance People independent Automation Process Improvements Future Avenues of Research

Think New

Leverage artificial intelligence

Simplify ( Reduce Complexity )

- Adoption

- Use

- Maintenance

People independent

Automation Process Improvements

Questions & Answers

References http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502 http://books.nap.edu/openbook.php?record_id=10695&page=137 http://www.logigear.com/newsletter/test_automation_top_5_suggested_best_practices.asp http://findarticles.com/p/articles/mi_m3230/is_n12_v28/ai_19040722 http://www.itwales.com/998515.htm http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502

http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502

http://books.nap.edu/openbook.php?record_id=10695&page=137

http://www.logigear.com/newsletter/test_automation_top_5_suggested_best_practices.asp

http://findarticles.com/p/articles/mi_m3230/is_n12_v28/ai_19040722

http://www.itwales.com/998515.htm

http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502

Suggested Readings http://www.developsense.com/presentations/emotions%20and%20test%20oracles.ppt - Michael Bolton http://www.satisfice.com/blog/ - James Bach http://testertested.blogspot.com/ - Pradeep Soundarajan

http://www.developsense.com/presentations/emotions%20and%20test%20oracles.ppt - Michael Bolton

http://www.satisfice.com/blog/ - James Bach

http://testertested.blogspot.com/ - Pradeep Soundarajan

Thank you  [email_address]

Thank you 

[email_address]