Application & Script Independent Framework: The Need For Data Normalization ASIF Date:  Wednesday, 14 th  November, 2007 V...
<ul><li>Name: Raj </li></ul><ul><li>Indian citizen </li></ul><ul><li>25 years </li></ul><ul><li>First time in US </li></ul...
Research
Past Research <ul><li>Iterative Automation Process using RUP </li></ul><ul><li>ART (Automation Review Tool ) </li></ul><ul...
Big  Question
Agenda <ul><li>Industry Quotes </li></ul><ul><li>Problems </li></ul><ul><li>Solution </li></ul><ul><li>Architecture  </li>...
Survey “ Requirement Changes are inevitable”
“ In QAI survey w hen asked, “ To what degree are the test products maintained and reused ?” 33% said always,  59% sometim...
“ requirements; the  real joke  is that the requirements end up being what the source code does.” “ 62 %  of respondents s...
Available Automation Frameworks
What’s Available <ul><li>Record / Playback </li></ul><ul><li>Data Driven Framework </li></ul><ul><li>Modular Architecture ...
Low ROI & High Maintenance  Effort ITERATIVE DEVELOPMENT REDUNDANCY INCONSISTENCY PROBLEM: CAUSES OF AUTOMATION FAILURE Fr...
Problem #1: Redundancy <ul><li>Scenario: </li></ul><ul><li>Test Data Sets : 500 Tables / Sheets </li></ul><ul><li>Test Cas...
Problem #1: Redundancy <ul><li>Future Release Changes </li></ul><ul><li>  - Object Type </li></ul><ul><li>  - Object Name ...
Problem 2#: Inconsistency <ul><li>Future Release Changes </li></ul><ul><li>  - Object Type </li></ul><ul><li>  - Object Na...
Problem #3: Evolving Development Models <ul><li>Agile Modeling </li></ul><ul><li>SCRUM </li></ul><ul><li>XP </li></ul><ul>...
Problem #3: Evolving Development Models <ul><li>Evolutionary Dev Models can handle changes gracefully and are becoming bet...
Solution
Suggested Solution : ASIF <ul><li>Data Normalization </li></ul><ul><li>User defined  Views </li></ul><ul><li>Other  RDBMS ...
Solution #1: Normalization <ul><li>Minimize duplication by storing only ONE copy of the  Test Data  </li></ul><ul><ul><li>...
Solution #2: User Defined Views <ul><li>1 View ~ 1 Test Case </li></ul><ul><li>View = Table 1 + Table 2 + .. (JOIN) </li><...
Solution #3: Reusable Library <ul><li>ASIF Library </li></ul><ul><ul><li>Talks with Views NOT Tables </li></ul></ul><ul><u...
Solution #4: Over Simplified Scripts <ul><li>Scripts </li></ul><ul><ul><li>Invoke ASIF Reusable Library </li></ul></ul><ul...
Implementation
ASIF Automation Architecture Name Age Views Tables Scripts
Implementation: Automation Process Setup
Implementation: Database Setup
Implementation: Automation Test Tool
General Flow PK Object Name / Type  Object Name / Type  Object Name / Type  0R Userid~EditBox UPass~EditBox UName~EditBox ...
Database Design User Id  Password  XYZ Login Page  OK Cancel User  name  :  Age  : XYZ User Information Raj 25 PK Object N...
Example #1 – Change in Requirements <ul><li>Requirement changes the business flow or logic </li></ul><ul><li>Updated only ...
Example #2 – Change in Design/Code <ul><li>Object name / Object type is added / changed by a developer  </li></ul><ul><li>...
Cost
Cost – Manual Testing <ul><li>Cost of manual Testing </li></ul><ul><li>=  time to develop test cases  </li></ul><ul><li>  ...
Cost- Automation using ASIF <ul><li>Cost of automation Testing </li></ul><ul><li>=  price of hardware </li></ul><ul><li>+ ...
Return on Investment
ASIF Return on Investment <ul><li>ROI  =  (cost of manual  –  cost of automation)  /  cost of automation </li></ul>ASIF gi...
Benefits
Benefits <ul><li>Easy maintainability </li></ul><ul><li>Test Data Storage and Retrieval Efficiency </li></ul><ul><li>Bette...
ASIF Break-even Point Test Runs Cost of  Testing n En = (Va + n * Da )/ (Vm + n * Dm ) V -  Expenditure for Implementation...
Benefits <ul><li>Accelerated time to Market </li></ul><ul><li>Abstract and decouple test data from scripts </li></ul><ul><...
Effort  Reduction Releases Effort Application Complexity  - Test Effort Reduction over Releases
Overall Performance Total Effort (%) Complexity Total  Effort Reduction (%)
Why ASIF and not any other automation framework ? <ul><li>Almost 90 % reduction in the script code </li></ul><ul><li>Handl...
Conclusion: When to Use ? <ul><li>Requirements are Dynamic </li></ul><ul><li>Ideal for  Iterative  /  Agile  /  SCRUM  Mod...
Future Avenues of Research
<ul><li>Think New  </li></ul><ul><li>Leverage artificial intelligence </li></ul><ul><li>Simplify ( Reduce Complexity ) </l...
Questions & Answers
References <ul><li>http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.htm...
Suggested Readings <ul><li>http://www.developsense.com/presentations/emotions%20and%20test%20oracles.ppt  -  Michael Bolto...
<ul><li>Thank you   </li></ul><ul><li>[email_address] </li></ul>
Upcoming SlideShare
Loading in …5
×

Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization

4,006 views

Published on

Published in: Technology
2 Comments
8 Likes
Statistics
Notes
  • super....
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • cool !

    can u send it to ajophilip@gmail.com?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
4,006
On SlideShare
0
From Embeds
0
Number of Embeds
148
Actions
Shares
0
Downloads
0
Comments
2
Likes
8
Embeds 0
No embeds

No notes for slide
  • Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization

    1. 1. Application & Script Independent Framework: The Need For Data Normalization ASIF Date: Wednesday, 14 th November, 2007 Venue: Orlando, Florida Presenter: Raj Kamal Company: Microsoft, India
    2. 2. <ul><li>Name: Raj </li></ul><ul><li>Indian citizen </li></ul><ul><li>25 years </li></ul><ul><li>First time in US </li></ul><ul><li>Flew over 7718 + miles to attend QAI </li></ul><ul><li>Basically from a village in Haryana , India. </li></ul><ul><li>Sports – Ping Pong, Cricket and Pool </li></ul><ul><li>Like learning new technologies and gadgets </li></ul>Who am I ? Born in Delhi Work @ MS Hyderabad
    3. 3. Research
    4. 4. Past Research <ul><li>Iterative Automation Process using RUP </li></ul><ul><li>ART (Automation Review Tool ) </li></ul><ul><li>Mapping RUP to Rational Testing Tools </li></ul><ul><ul><li>Presented at http://ifstep.org/ </li></ul></ul><ul><li>ROI of Testing </li></ul>http://geektester.blogspot.com
    5. 5. Big Question
    6. 6. Agenda <ul><li>Industry Quotes </li></ul><ul><li>Problems </li></ul><ul><li>Solution </li></ul><ul><li>Architecture </li></ul><ul><li>Implementation </li></ul><ul><li>Benefits </li></ul>
    7. 7. Survey “ Requirement Changes are inevitable”
    8. 8. “ In QAI survey w hen asked, “ To what degree are the test products maintained and reused ?” 33% said always, 59% sometimes, and 8% never ” ” Only 19.4% of respondents said they used some type of automated testing tool, the remainder relying upon manual testing methods.” “ The prevailing reasons given for not automating test processes were: Time (35%); cost (33%); script/data maintenance (26%); and complexity (18%) “ Survey: Big Spends, Little Comfort
    9. 9. “ requirements; the real joke is that the requirements end up being what the source code does.” “ 62 % of respondents stated they could be driving a new Mercedes  or retired if they had a dollar ($) for every time they had to rewrite a test case because requirements changes weren't effectively communicated &quot;it is often the case that development can slip on their timelines, but testing cannot.“ Survey: Automation is everywhere
    10. 10. Available Automation Frameworks
    11. 11. What’s Available <ul><li>Record / Playback </li></ul><ul><li>Data Driven Framework </li></ul><ul><li>Modular Architecture </li></ul><ul><li>Keyword Driven Framework </li></ul><ul><li>Hybrid Framework </li></ul><ul><li>http://en.wikipedia.org/wiki/Test_Automation_Framework </li></ul>
    12. 12. Low ROI & High Maintenance Effort ITERATIVE DEVELOPMENT REDUNDANCY INCONSISTENCY PROBLEM: CAUSES OF AUTOMATION FAILURE Frequent updates Multiple versions Difficult to Debug Common Test Data Automation Users Complexity Grows Change Requests Phases Implementation Frequent Script updates SKILLS People Issues Common Flow Frequent Failures Coding
    13. 13. Problem #1: Redundancy <ul><li>Scenario: </li></ul><ul><li>Test Data Sets : 500 Tables / Sheets </li></ul><ul><li>Test Cases Covered: 1000 </li></ul><ul><li>Testers: 4 </li></ul><ul><li>Modules: 4 </li></ul>Unintentional Repetition of Test Data & Multiple copies of the same data maintained
    14. 14. Problem #1: Redundancy <ul><li>Future Release Changes </li></ul><ul><li> - Object Type </li></ul><ul><li> - Object Name </li></ul><ul><li> - Object Data </li></ul><ul><li>Impact / Loss [ Domino effect ] </li></ul><ul><li> - Multiple Script Failure </li></ul><ul><li> - Rework updating multiple instances </li></ul><ul><li> - Loss of Data Integrity </li></ul><ul><li> </li></ul>
    15. 15. Problem 2#: Inconsistency <ul><li>Future Release Changes </li></ul><ul><li> - Object Type </li></ul><ul><li> - Object Name </li></ul><ul><li> - Object Data </li></ul><ul><li>Impact / Loss </li></ul><ul><li>Inconsistency – Different versions of Data </li></ul><ul><li>Tedious Process </li></ul><ul><li>Misleading Result / Invalid Bugs </li></ul>
    16. 16. Problem #3: Evolving Development Models <ul><li>Agile Modeling </li></ul><ul><li>SCRUM </li></ul><ul><li>XP </li></ul><ul><li>RUP </li></ul><ul><li>Iterative Development </li></ul><ul><li>Microsoft Solutions Framework </li></ul><ul><li>And Many More ….. </li></ul>
    17. 17. Problem #3: Evolving Development Models <ul><li>Evolutionary Dev Models can handle changes gracefully and are becoming better every day…. </li></ul>But Is our automation evolving / adapting with the same pace ?
    18. 18. Solution
    19. 19. Suggested Solution : ASIF <ul><li>Data Normalization </li></ul><ul><li>User defined Views </li></ul><ul><li>Other RDBMS concepts </li></ul><ul><li>100 % Reusable Plug and Play Library </li></ul><ul><li>Simplified Scripts (just to invoke library) </li></ul>“ ASIF, just like Rome, wasn't built in a day”
    20. 20. Solution #1: Normalization <ul><li>Minimize duplication by storing only ONE copy of the Test Data </li></ul><ul><ul><li>Object Type </li></ul></ul><ul><ul><li>Object Name </li></ul></ul><ul><ul><li>Object Data </li></ul></ul>Have Test Data in Normal Form – 1 NF, 2 NF, 3 NF Eliminate Redundancy, Inconsistency
    21. 21. Solution #2: User Defined Views <ul><li>1 View ~ 1 Test Case </li></ul><ul><li>View = Table 1 + Table 2 + .. (JOIN) </li></ul><ul><li>View contain “FLOW” of Test Execution </li></ul><ul><li>Table Updated ~ Views get updated </li></ul><ul><li>Views get Updates ~ Tables get updated </li></ul><ul><li>Easy to create / drop / modify </li></ul><ul><li>Contains BUSINESS LOGIC </li></ul>
    22. 22. Solution #3: Reusable Library <ul><li>ASIF Library </li></ul><ul><ul><li>Talks with Views NOT Tables </li></ul></ul><ul><ul><li>Can Read / Understand / Update Views </li></ul></ul><ul><ul><li>Perform ACTION / VERIFICATION </li></ul></ul><ul><ul><li>Handles Errors & Exceptions </li></ul></ul><ul><ul><li>Can write logs / results (PASS / FAIL) </li></ul></ul><ul><ul><li>100 % Reusable – DON’T CONTAIN BUSINESS LOGIC </li></ul></ul><ul><ul><li>Interface between Scripts & Test Data </li></ul></ul>
    23. 23. Solution #4: Over Simplified Scripts <ul><li>Scripts </li></ul><ul><ul><li>Invoke ASIF Reusable Library </li></ul></ul><ul><ul><li>Passes “Test Case Id” as Parameter </li></ul></ul><ul><ul><li>DON’T CONTAIN BUSINESS LOGIC </li></ul></ul><ul><ul><li>DON’T CONTAIN ACTION CODE </li></ul></ul><ul><ul><li>DON’T CONTAIN VERIFICATION CODE </li></ul></ul><ul><ul><li>90 % Reduction in LOC (Lines of Code) </li></ul></ul>
    24. 24. Implementation
    25. 25. ASIF Automation Architecture Name Age Views Tables Scripts
    26. 26. Implementation: Automation Process Setup
    27. 27. Implementation: Database Setup
    28. 28. Implementation: Automation Test Tool
    29. 29. General Flow PK Object Name / Type Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox UName~EditBox TC_01 User1 Pwd1 Raj TC_02 User2 Pwd2 Sam
    30. 30. Database Design User Id Password XYZ Login Page OK Cancel User name : Age : XYZ User Information Raj 25 PK Object Name / Type Object Name / Type 0R Userid~EditBox UPass~EditBox TC_01 User1 Pwd1 TC_02 User2 Pwd2 PK Object Name / Type Object Name / Type 0R UName~EditBox UAge~EditBox TC_01 Raj 25 TC_02 Sam 28 PK Object Name / Type Object Name / Type 0R Userid~EditBox UName~EditBox TC_01 User1 Raj TC_02 User2 Sam
    31. 31. Example #1 – Change in Requirements <ul><li>Requirement changes the business flow or logic </li></ul><ul><li>Updated only views written for that test case </li></ul><ul><li>Tables get updated automatically </li></ul><ul><li>Scripts need not to be touched. </li></ul><ul><li>Run scripts and validate the changes </li></ul>
    32. 32. Example #2 – Change in Design/Code <ul><li>Object name / Object type is added / changed by a developer </li></ul><ul><li>Updated base table only at one place </li></ul><ul><li>Views get updated automatically </li></ul><ul><li>Scripts need not to be touched. </li></ul><ul><li>Run scripts and validate the changes </li></ul>
    33. 33. Cost
    34. 34. Cost – Manual Testing <ul><li>Cost of manual Testing </li></ul><ul><li>= time to develop test cases </li></ul><ul><li>    + time to maintain test cases x number of times tests are executed </li></ul><ul><li>   + time to execute manual testing x number of times tests are executed </li></ul>
    35. 35. Cost- Automation using ASIF <ul><li>Cost of automation Testing </li></ul><ul><li>= price of hardware </li></ul><ul><li>+ price of software </li></ul><ul><li>   + time to develop scripts </li></ul><ul><li>   +   time to maintain scripts x number of times scripts are executed </li></ul><ul><li>   + time to execute scripts x number of times scripts are executed </li></ul>
    36. 36. Return on Investment
    37. 37. ASIF Return on Investment <ul><li>ROI = (cost of manual – cost of automation) / cost of automation </li></ul>ASIF gives ROI of 72 % over the course of the year
    38. 38. Benefits
    39. 39. Benefits <ul><li>Easy maintainability </li></ul><ul><li>Test Data Storage and Retrieval Efficiency </li></ul><ul><li>Better test coverage </li></ul><ul><li>Modularization </li></ul><ul><li>Gracefully handles scalability </li></ul>
    40. 40. ASIF Break-even Point Test Runs Cost of Testing n En = (Va + n * Da )/ (Vm + n * Dm ) V - Expenditure for Implementation D - Expenditure for single test execution
    41. 41. Benefits <ul><li>Accelerated time to Market </li></ul><ul><li>Abstract and decouple test data from scripts </li></ul><ul><li>Provides 100 % Reusable Plug & Play Library </li></ul><ul><li>Reduced cost of quality </li></ul><ul><li>Decouple complex business function </li></ul>
    42. 42. Effort Reduction Releases Effort Application Complexity - Test Effort Reduction over Releases
    43. 43. Overall Performance Total Effort (%) Complexity Total Effort Reduction (%)
    44. 44. Why ASIF and not any other automation framework ? <ul><li>Almost 90 % reduction in the script code </li></ul><ul><li>Handles requirement changes gracefully </li></ul><ul><li>ASIF Automation framework don’t take breaks, don’t need to eat, don’t take calls from their family  </li></ul>
    45. 45. Conclusion: When to Use ? <ul><li>Requirements are Dynamic </li></ul><ul><li>Ideal for Iterative / Agile / SCRUM Model </li></ul><ul><li>At least 3 Test Runs to obtain ROI </li></ul><ul><li>Testers lack Coding skills </li></ul><ul><li>Ideal to run Regression and BVT Suite </li></ul><ul><li>Permanent long term solution for Client </li></ul>
    46. 46. Future Avenues of Research
    47. 47. <ul><li>Think New </li></ul><ul><li>Leverage artificial intelligence </li></ul><ul><li>Simplify ( Reduce Complexity ) </li></ul><ul><li> - Adoption </li></ul><ul><li> - Use </li></ul><ul><li>- Maintenance </li></ul><ul><li>People independent </li></ul><ul><li>Automation Process Improvements </li></ul>Future Avenues of Research
    48. 48. Questions & Answers
    49. 49. References <ul><li>http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html </li></ul><ul><li>http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502 </li></ul><ul><li>http://books.nap.edu/openbook.php?record_id=10695&page=137 </li></ul><ul><li>http://www.logigear.com/newsletter/test_automation_top_5_suggested_best_practices.asp </li></ul><ul><li>http://findarticles.com/p/articles/mi_m3230/is_n12_v28/ai_19040722 </li></ul><ul><li>http://www.itwales.com/998515.htm </li></ul><ul><li>http://www.borland.com/us/company/news/press_releases/2006/10_31_06_borland_qa_professionals_survey.html </li></ul><ul><li>http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=8502 </li></ul>
    50. 50. Suggested Readings <ul><li>http://www.developsense.com/presentations/emotions%20and%20test%20oracles.ppt - Michael Bolton </li></ul><ul><li>http://www.satisfice.com/blog/ - James Bach </li></ul><ul><li>http://testertested.blogspot.com/ - Pradeep Soundarajan </li></ul>
    51. 51. <ul><li>Thank you  </li></ul><ul><li>[email_address] </li></ul>

    ×