ANZTB (Melbourne) - Hybrid Keyword Data Driven Automation Frameworks - Track

7,198 views
7,012 views

Published on

0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,198
On SlideShare
0
From Embeds
0
Number of Embeds
651
Actions
Shares
0
Downloads
0
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Automation Frameworks by Jonathon Wright Tuesday, 2 nd March 2010 ANZTB 2010
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • ANZTB 2010 – Conference Hybrid Keyword Data Driven Frameworks by Jonathon Lee Wright Tuesday, 2 nd March 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • Automation Development Services www.automation.org.uk Wednesday 27 January 2010 ADS 2003
  • ANZTB (Melbourne) - Hybrid Keyword Data Driven Automation Frameworks - Track

    1. 1. Hybrid Keyword Data Driven Automation Frameworks An Insight to Approaches, Techniques & Methodologies. Jonathon Lee Wright [email_address] www.automation.org.uk www.linkedin.com/in/automation 2nd March 2010
    2. 2. Hybrid Keyword Data Driven Automation Frameworks <ul><li>What does this mean to you now? </li></ul>Hybrid Keyword Data Driven Automation Frameworks (Slide 1) 2nd March 2010 Introduction
    3. 3. Presentation Plan <ul><li>Background </li></ul>2nd March 2010 Introduction Hybrid Keyword Data Driven Automation Frameworks (Slide 2) Traditional Approach Automation: Timeline Automation: Failure Rate Reasons for Past Failure Understanding Failure Break Even Point Where to Start GAP Analysis Initial Investment Break Even Point Framework: Advantages Which Test Tool One Dedicated Team Business Process Testing Keyword Driven Data Driven Modularity Driven Summary Framework Approach (Hybrid Keyword Data Driven Automation Frameworks)
    4. 4. Moving swiftly past the hype <ul><li>Historically Automation is perceived as a “Silver Bullet” of the Testing world. </li></ul><ul><li>“ The term has been adopted into a general metaphor, where &quot;silver bullet&quot; refers to any straight forward solution perceived to have extreme effectiveness . The phrase typically appears with an expectation that some new technology or practice will easily cure a major prevailing problem ” </li></ul>Introduction 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 4)
    5. 5. Timeline: Product Test Automation <ul><li>Historical trends in automation frameworks: </li></ul>Background Hybrid Keyword Data Driven Automation Frameworks (Slide 5)
    6. 6. Learning from past experience Background 2nd March 2010 92% FAIL TO MEET TARGET ROI Hybrid Keyword Data Driven Automation Frameworks (Slide 6) Source A: ROI on Test Automation - http://www.keane.com Source B: HP pay 4.5billion to acquire mercury - http://www.nytimes.com/2006/07/26/technology/26hewlett.html 2004 2010 (Estimated) Industry: Test Automation (net worth) $1 billion $6.3 billion Automation Projects (failure cost) $0.6 billion $3.8 billion
    7. 7. Reasons for past failure <ul><li>Lack of defined automation methodology </li></ul>Traditional Approach 2nd March 2010 WHY DO AUTOMATION PROJECTS TYPICALLY FAIL? ATLM Diagram: Automated Software Testing (Dustin – Rashka – Paul) - ISBN 0-201-43287-0 Source: Maximizing ROI and Avoiding the Pitfalls of Test Automation (Bill Hayduk - RTTS) - http://www.rttsweb.com/research <ul><li>Automation is not treated as a legitimate project with the necessary planning / resources </li></ul>Hybrid Keyword Data Driven Automation Frameworks (Slide 7)
    8. 8. Reasons for past failure <ul><li>Test Automation is typically performed at the end of the development cycle </li></ul>Traditional Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 8) User Requirements System Requirements Global Design Detailed Design Implementation Testing <ul><li>Traditional - Test Automation </li></ul><ul><li>Functional Test Tool Dependences: </li></ul><ul><li>Application Under Test (AUT) Delivery </li></ul><ul><li>- Learn Object Repository </li></ul><ul><li>Hybrid - Test Automation </li></ul><ul><li>Business Process Modelling (BPM) </li></ul><ul><li>- Mapping Business Process </li></ul><ul><li>Business Process Testing (BPT) - Reusable Modules (Jigsaw Pieces) </li></ul><ul><li>Descriptive Programming (DP) - Enables Dynamic Object Repository </li></ul>
    9. 9. Reasons for past failure <ul><li>After the initial success the automation scripts are not maintained for future builds </li></ul><ul><li>(contributing to a high failure rate of the remaining 32% of working automation projects) </li></ul><ul><li>Testers are typically untrained in test tools and programming techniques </li></ul><ul><li>No modularization ( reusable functions ) in automation scripts </li></ul><ul><li>Automated tests cases are usually designed based on front end functionality ( black box testing ) </li></ul>Traditional Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 9)
    10. 10. Putting the pieces of the puzzle together Real World Example – Simple Login Screen LOGIN Traditional Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 10) Login Register Reset Password Registered Registration Forgot Password Login Reset Password Black-Box Activity Diagram
    11. 11. Understanding why automation fails <ul><li>The Mine Sweeper Effect </li></ul>Traditional Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 11) Automated testing repeats the same route over and over. Initially any defects that are found are whilst developing the scripts (i.e. manually) Subsequent executions will have little chance of ever finding further defects unless the flow changes
    12. 12. Record Replay is not your friend <ul><li>Linear process mapping (AUT) </li></ul>Traditional Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 12) Video Source: YouTube - Bugatti Veyron Lake Crash - http://www.youtube.com/watch?v=4NJmB1F2mdE
    13. 13. Breaking Even Point Traditional Approach 2nd March 2010 Equation Source: Automated Software Testing (Dustin – Rashka – Paul) - ISBN 0-201-43287-0 Hybrid Keyword Data Driven Automation Frameworks (Slide 13) Preparation ( V ) Execution ( D ) ROI Resource ( R ) for (n) Automated Tests Test Manual Automated Manual Automated 1 5 10 20 Scenario 1 30 60 11 1.1   3 149% 77% 50% 32% Scenario 2 30 60 11 1.1   3 149% 77% 50% 32% Scenario 3 30 60 9 0.9   4 156% 86% 57% 37% Scenario 4 30 60 9 0.9   4 156% 86% 57% 37% Scenario 5 30 60 10 1   4 153% 81% 54% 35% Scenario 6 30 60 10 1   4 153% 81% 54% 35% Scenario 7 30 60 15 1.5   3 137% 64% 42% 27% Scenario 8 30 60 30 3   2 105% 42% 27% 19% Scenario 9 30 60 22 2.2   2 120% 51% 33% 22% Scenario 10 30 60 12 1.2   3 146% 73% 48% 31% Totals 5 hrs 10 hrs 140 mins 14 mins   3.2 142% 72% 47% 31% R n = A a / A m = (V a + n *D a ) / (V m + n *D m )
    14. 14. Framework Approach (Hybrid Keyword Data Driven Automation Frameworks) Introducing Hybrid Keyword Data Driven Automation Frameworks (Slide 14) 2nd March 2010 Automation: Timeline Automation: Failure Rate Reasons for Past Failure Understanding Failure Break Even Point Where to Start GAP Analysis Initial Investment Break Even Point Framework: Advantages Which Test Tool One Dedicated Team Business Process Testing Keyword Driven Data Driven Modularity Driven Summary
    15. 15. Where to start? <ul><li>Remember to think of each module as a individual Jigsaw piece but keep in mind what is needed to complete the entire puzzle </li></ul>Framework Approach 2nd March 2010 “ Under promise, Over deliver?” Quick wins should be avoided “ Start SMALL think BIG” NEVER expect to automate 100% Focus on key critical business processes Keep it simple, wherever possible Hybrid Keyword Data Driven Automation Frameworks (Slide 15) First find out Then you can work out What needs to be tested? What needs be automated? What can be tested? What can be automated? What could be tested? What could be automated?
    16. 16. <ul><li>Understanding the full extent of Application Under Test (AUT): </li></ul>GAP Analysis Framework Approach 2nd March 2010 Business Risk Complexity Usage Feasibility Reusability GAP analysis = complete application functionality – actual “core” business usage Hybrid Keyword Data Driven Automation Frameworks (Slide 16) NOTE : The GAP analysis file can be used to generate the parameter file which in turn is used to generate the dynamic object repository .
    17. 17. Investing the time <ul><li>Allocate realistic time frames & resources to achieve the initial framework: </li></ul>Framework Approach 2nd March 2010 Framework = planning + design + development + testing Hybrid Keyword Data Driven Automation Frameworks (Slide 17) based on sample set of 10 scenarios leverage leverage initial investment Framework 20hrs (prep) 14mins (execution) ROI = 2.8 Manual 5hrs (prep) 140mins (execution) N/A Traditional 10hrs (prep) 14mins (execution) ROI = 3.2
    18. 18. Breaking Even Point Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 18) Preparation ( V ) Execution ( D ) ROI Resource ( R ) for (n) Automated Tests Test Manual Automated Manual Automated 1 5 10 20 Scenario 1 30 120 11 1.1 3 295% 34% 16% 11% Scenario 2 30 120 11 1.1 3 295% 34% 16% 11% Scenario 3 30 120 9 0.9 3  310% 38% 17% 11% Scenario 4 30 120 9 0.9 3  310% 38% 17% 11% Scenario 5 30 120 10 1 3  302% 36% 16% 11% Scenario 6 30 120 10 1 3  302% 36% 16% 11% Scenario 7 30 120 15 1.5 3  270% 30% 15% 10% Scenario 8 30 120 30 3 2  205% 21% 12% 10% Scenario 9 30 120 22 2.2 2  235% 25% 13% 10% Scenario 10 30 120 12 1.2 3  288% 33% 16% 11% Totals 5 hrs 20 hrs 140 mins 14 mins   2.8 281% 33% 16% 11% R n = A a / A m = (V a / n + n *D a ) / (V m + n *D m )
    19. 19. Key Advantages Framework Approach 2nd March 2010 <ul><li>Maintainability – significantly reduces the test maintenance effort </li></ul><ul><li>Reusability – due to modularity of test cases and library functions </li></ul><ul><li>Manageability - effective test design, execution, and traceability </li></ul><ul><li>Accessibility – to design, develop & modify tests whilst executing </li></ul><ul><li>Availability – scheduled execution can run unattended on a 24/7 basis </li></ul><ul><li>Reliability – due to advanced error handling and scenario recovery </li></ul><ul><li>Flexibility – framework independent of AUT or environment </li></ul><ul><li>Measurability – customisable reporting of test results ensure quality </li></ul>Hybrid Keyword Data Driven Automation Frameworks (Slide 19)
    20. 20. Which? <ul><li>Which Automation Framework Suits You? </li></ul>Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 20) * optimised solution with preferred vendors Approach BPT/BPM Keyword Data Modularity Example Functional Test Tool   QuickTest Professional Test Management Tool   Quality Center (BPTEE, BPM) Standalone IDE Tool  VBS (Windows Scripting Host) Combination of all the above     Hybrid Solution*
    21. 21. One “dedicated” Team Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 21)
    22. 22. Business Process Testing (BPT) Collaborative Solution Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 22) Login Subject Matter Experts (SME) design tests using documentation to map logical data flow and process modelling Step 2 – Design Test Case(s) Input Field Values Screen Format Username 64 Characters Text Password 12 Characters Hidden Login Button Visible Subject Matter Experts (SME) System Requirements Specification (SRS) Functional specifications document (FSD) Software design document (SDD) Detailed design document (DDD) Step 1 – Export Core Logic Framework Automation Experts Step 3 – Code Generation Framework Automation Experts transform designed tests into automation components ready for keyword stage
    23. 23. Keyword Driven Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 23) Login Subject Matter Experts (SME) Step 4 – Create Test Case(s) Subject Matter Experts (SME) create test cases using business level keywords stored within the MasterDriver file containing all the test scenarios relating to the AUT Action Object Event(Type) Activate Login Window Set Username Editbox Set Password Editbox Press Login Button Generic Reusable Libraries Example Runtime Process The framework calls the necessary functions to process each keyword / component and dynamically creates the object repository at run-time <ul><li>Log.Framework </li></ul><ul><li>ObjectRepository.CreateObject.Window </li></ul><ul><li>ObjectRepository.CreateObject.Button </li></ul><ul><li>ClearAlerts </li></ul><ul><li>Log.Events </li></ul><ul><li>Sync.WinWaitObject.Button </li></ul><ul><li>Action.Preamble.Check.ButtonState </li></ul><ul><li>Event.PushDynamic.Button </li></ul><ul><li>Action.Postamble.Check.ButtonState </li></ul><ul><li>Action.Reaction.Check </li></ul><ul><li>Popup.Handler </li></ul><ul><li>Error.Recovery (retry/skip/baseline) </li></ul><ul><li>Log.Results </li></ul><ul><li>Log.Reports </li></ul>Driver & Data Repository MasterDriver File Parameters File The Hybrid Keyword Data Driven Automation Framework processes each action/object/event keyword/component in turn
    24. 24. Data Driven Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 24) Step 5 – Create Test Data Login Subject Matter Experts (SME) create test data within the MasterDriver file from a number of possible input data sources Subject Matter Experts (SME) Test Object Expected Result Test Data Source Username Pass DB Query Password Pass Unique LoginFlag Fail Negative Input Data Sources
    25. 25. Examples of working smarter Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 25) Login Register Reset Password Terms & Conditions Introduction of new functionality Change to current functionality
    26. 26. Guidelines behind working smarter <ul><li>Centralized - creation of the centralized master driver file to provide stable , repeatable keyword / data driven test scenarios </li></ul><ul><li>Dynamic - runtime generation of the object repository from the parameter file utilizing descriptive programming techniques </li></ul><ul><li>Legacy support - for manual execution & AUT roll back </li></ul><ul><li>Version controlled - code , driver files & object repositories </li></ul><ul><li>Unified reporting - supporting real-time test execution & generic test results output (.xml) to analyze quality of builds </li></ul><ul><li>Generic reusable modules / libraries - provide reliable processing of preamble and postamble actions </li></ul>Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 26)
    27. 27. Modularity: Generic Framework Architecture Framework Approach 2nd March 2010 Hybrid Keyword Data Driven Automation Frameworks (Slide 27)
    28. 28. S UMMARY Framework Approach 2nd March 2010 <ul><li>Hybrid – utilising the best technologies and resources to do the job </li></ul><ul><li>Keyword – creating simple & robust test cases written using business level keywords combined with natural language </li></ul><ul><li>Data – effective use of business data to provide as an input source </li></ul><ul><li>Driven – processes component actions , objects & events seamlessly </li></ul><ul><li>Automation – that is collaborative , distributed and scalable </li></ul><ul><li>Frameworks – independent of application or environment under test </li></ul>Hybrid Keyword Data Driven Automation Frameworks (Slide 28)
    29. 29. Hybrid Keyword Data Driven Automation Frameworks What does it mean to you now? Thanks for your attention!

    ×