Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization

Application & Script Independent Framework: The Need For Data Normalization ASIF Date: Wednesday, 14 th November, 2007 Venue: Orlando, Florida Presenter: Raj Kamal Company: Microsoft, India

Name: Raj
Indian citizen
25 years
First time in US
Flew over 7718 + miles to attend QAI
Basically from a village in Haryana , India.
Sports – Ping Pong, Cricket and Pool
Like learning new technologies and gadgets
Who am I ? Born in Delhi Work @ MS Hyderabad

Research

Past Research
Iterative Automation Process using RUP
ART (Automation Review Tool )
Mapping RUP to Rational Testing Tools
Presented at http://ifstep.org/
ROI of Testing
http://geektester.blogspot.com

Big Question

Agenda
Industry Quotes
Problems
Solution
Architecture
Implementation
Benefits

Survey “ Requirement Changes are inevitable”

“ In QAI survey w hen asked, “ To what degree are the test products maintained and reused ?” 33% said always, 59% sometimes, and 8% never ” ” Only 19.4% of respondents said they used some type of automated testing tool, the remainder relying upon manual testing methods.” “ The prevailing reasons given for not automating test processes were: Time (35%); cost (33%); script/data maintenance (26%); and complexity (18%) “ Survey: Big Spends, Little Comfort

“ requirements; the real joke is that the requirements end up being what the source code does.” “ 62 % of respondents stated they could be driving a new Mercedes  or retired if they had a dollar ($) for every time they had to rewrite a test case because requirements changes weren't effectively communicated "it is often the case that development can slip on their timelines, but testing cannot.“ Survey: Automation is everywhere

Available Automation Frameworks

What’s Available
Record / Playback
Data Driven Framework
Modular Architecture
Keyword Driven Framework
Hybrid Framework
http://en.wikipedia.org/wiki/Test_Automation_Framework

Low ROI & High Maintenance Effort ITERATIVE DEVELOPMENT REDUNDANCY INCONSISTENCY PROBLEM: CAUSES OF AUTOMATION FAILURE Frequent updates Multiple versions Difficult to Debug Common Test Data Automation Users Complexity Grows Change Requests Phases Implementation Frequent Script updates SKILLS People Issues Common Flow Frequent Failures Coding

Problem #1: Redundancy
Scenario:
Test Data Sets : 500 Tables / Sheets
Test Cases Covered: 1000
Testers: 4
Modules: 4
Unintentional Repetition of Test Data & Multiple copies of the same data maintained

Problem #1: Redundancy
Future Release Changes
- Object Type
- Object Name
- Object Data
Impact / Loss [ Domino effect ]
- Multiple Script Failure
- Rework updating multiple instances
- Loss of Data Integrity

Problem 2#: Inconsistency
Future Release Changes
- Object Type
- Object Name
- Object Data
Impact / Loss
Inconsistency – Different versions of Data
Tedious Process
Misleading Result / Invalid Bugs

Problem #3: Evolving Development Models
Agile Modeling
SCRUM
XP
RUP
Iterative Development
Microsoft Solutions Framework
And Many More …..

Problem #3: Evolving Development Models
Evolutionary Dev Models can handle changes gracefully and are becoming better every day….
But Is our automation evolving / adapting with the same pace ?

Solution

Suggested Solution : ASIF
Data Normalization
User defined Views
Other RDBMS concepts
100 % Reusable Plug and Play Library
Simplified Scripts (just to invoke library)
“ ASIF, just like Rome, wasn't built in a day”

Solution #1: Normalization
Minimize duplication by storing only ONE copy of the Test Data
Object Type
Object Name
Object Data
Have Test Data in Normal Form – 1 NF, 2 NF, 3 NF Eliminate Redundancy, Inconsistency

Solution #2: User Defined Views
1 View ~ 1 Test Case
View = Table 1 + Table 2 + .. (JOIN)
View contain “FLOW” of Test Execution
Table Updated ~ Views get updated
Views get Updates ~ Tables get updated
Easy to create / drop / modify
Contains BUSINESS LOGIC

Solution #3: Reusable Library
ASIF Library
Talks with Views NOT Tables
Can Read / Understand / Update Views
Perform ACTION / VERIFICATION
Handles Errors & Exceptions
Can write logs / results (PASS / FAIL)
100 % Reusable – DON’T CONTAIN BUSINESS LOGIC
Interface between Scripts & Test Data

Solution #4: Over Simplified Scripts
Scripts
Invoke ASIF Reusable Library
Passes “Test Case Id” as Parameter
DON’T CONTAIN BUSINESS LOGIC
DON’T CONTAIN ACTION CODE
DON’T CONTAIN VERIFICATION CODE
90 % Reduction in LOC (Lines of Code)

Implementation

ASIF Automation Architecture Name Age Views Tables Scripts

Implementation: Automation Process Setup

Implementation: Database Setup

Implementation: Automation Test Tool

General Flow PK Object Name / Type Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox UName~EditBox TC_01 User1 Pwd1 Raj TC_02 User2 Pwd2 Sam

Database Design User Id Password XYZ Login Page OK Cancel User name : Age : XYZ User Information Raj 25 PK Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox TC_01 User1 Pwd1 TC_02 User2 Pwd2 PK Object Name / Type Object Name / Type 0R UName~EditBox UAge~EditBox TC_01 Raj 25 TC_02 Sam 28 PK Object Name / Type Object Name / Type 0R Userid~EditBox UName~EditBox TC_01 User1 Raj TC_02 User2 Sam

Example #1 – Change in Requirements
Requirement changes the business flow or logic
Updated only views written for that test case
Tables get updated automatically
Scripts need not to be touched.
Run scripts and validate the changes

Example #2 – Change in Design/Code
Object name / Object type is added / changed by a developer
Updated base table only at one place
Views get updated automatically
Scripts need not to be touched.
Run scripts and validate the changes

Cost – Manual Testing
Cost of manual Testing
= time to develop test cases
+ time to maintain test cases x number of times tests are executed
+ time to execute manual testing x number of times tests are executed

Cost- Automation using ASIF
Cost of automation Testing
= price of hardware
+ price of software
+ time to develop scripts
+ time to maintain scripts x number of times scripts are executed
+ time to execute scripts x number of times scripts are executed

Return on Investment

ASIF Return on Investment
ROI = (cost of manual – cost of automation) / cost of automation
ASIF gives ROI of 72 % over the course of the year

Benefits

Benefits
Easy maintainability
Test Data Storage and Retrieval Efficiency
Better test coverage
Modularization
Gracefully handles scalability

ASIF Break-even Point Test Runs Cost of Testing n En = (Va + n * Da )/ (Vm + n * Dm ) V - Expenditure for Implementation D - Expenditure for single test execution

Benefits
Accelerated time to Market
Abstract and decouple test data from scripts
Provides 100 % Reusable Plug & Play Library
Reduced cost of quality
Decouple complex business function

Effort Reduction Releases Effort Application Complexity - Test Effort Reduction over Releases

Overall Performance Total Effort (%) Complexity Total Effort Reduction (%)

Why ASIF and not any other automation framework ?
Almost 90 % reduction in the script code
Handles requirement changes gracefully
ASIF Automation framework don’t take breaks, don’t need to eat, don’t take calls from their family 

Conclusion: When to Use ?
Requirements are Dynamic
Ideal for Iterative / Agile / SCRUM Model
At least 3 Test Runs to obtain ROI
Testers lack Coding skills
Ideal to run Regression and BVT Suite
Permanent long term solution for Client

Future Avenues of Research

Think New
Leverage artificial intelligence
Simplify ( Reduce Complexity )
- Adoption
- Use
- Maintenance
People independent
Automation Process Improvements
Future Avenues of Research

Questions & Answers

References
http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502
http://books.nap.edu/openbook.php?record_id=10695&page=137
http://www.logigear.com/newsletter/test_automation_top_5_suggested_best_practices.asp
http://findarticles.com/p/articles/mi_m3230/is_n12_v28/ai_19040722
http://www.itwales.com/998515.htm
http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502

Suggested Readings
http://www.developsense.com/presentations/emotions%20and%20test%20oracles.ppt - Michael Bolton
http://www.satisfice.com/blog/ - James Bach
http://testertested.blogspot.com/ - Pradeep Soundarajan

Thank you 
[email_address]

http://at4qa.blogspot.com	63
http://www.geektester.blogspot.com	60
http://geektester.blogspot.com	36
http://www.slideshare.net	11
http://www.at4qa.blogspot.com	2
http://at4qa.blogspot.in	1
http://at4qa.blogspot.com.br	1

Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization

by guest0efb5e on Sep 07, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 174

Statistics

Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization — Presentation Transcript