• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Susan windsor soft test   16th november 2005
 

Susan windsor soft test 16th november 2005

on

  • 870 views

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

Statistics

Views

Total Views
870
Views on SlideShare
789
Embed Views
81

Actions

Likes
0
Downloads
5
Comments
0

1 Embed 81

http://softtest.ie 81

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Susan windsor soft test   16th november 2005 Susan windsor soft test 16th november 2005 Presentation Transcript

    • Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD Title slide Strategic Direction for Functional Test Automation Soft Test 2005
    • AGENDA
      • Future for Functional Testing
      • History of Functional Automation
      • Strategic Direction
      • Automation Frameworks in Action
      • Impact upon You?
    • Future for Functional Testing
    • Increased Demand………..?
      • Global development, with large projects having multi site, multi geography and multi suppliers to contend with.
      • Corporate and regulatory requirements growing all the time
      • Business demands new applications, faster and cheaper to obtain competitive advantage
      • Closer alignment of IT and business to repair lack of business confidence
    • ..Or, increased pressures?
      • Automation frameworks and greater business knowledge requirements, mean reduced numbers of traditional functional testers
      • Agile development methods mean developers undertake more unit and component testing
      • Growth of outsourced testing to different geographies, leading to greater competition for the roles
      Is the role of the functional tester (who is neither technical or business specialist) dead?
    • History of Functional Automation
    • Record and Playback
      • “ all you need to do…….…!”
      • Easy to record the scripts
      • Extremely fragile
      • Expensive to maintain
      • On average, each test run requires at least 50% of the scripts to be recorded again
      Please tell me no one does this anymore!!!
    • Scripting
      • Use the scripting language to write scripts that do what you want
      • Build in as much robustness as possible
      • But, you’re building an application to test an application, which also needs testing!
      • Maintenance costs can be very high
      • Needs programmer skills
      • Cost of implementation not affordable at project level
    • Table Driven
      • Like scripting but more flexible and greater re-use
      • Remove items from the script that change, e.g. data
      • Reduces maintenance costs a bit
      • Implementation costs about the same
      • Still requires specialist skills to implement
    • Functional Test Automation is Broken!
      • Focus on technology rather than business needs
      • 80% of functional testing still manual
      • 60% to 70% of automation tools used for non-functional testing
      • Typically, traditional functional automation stops at 100 scripts, regardless of test coverage requirement
      • Critical factors; cost of implementation and maintenance, skills required and asset sharing over different technologies
    • Strategic Direction
    • Automation Frameworks
      • Wouldn’t it be good if……..
        • Tests could be documented in common format, regardless of whether they are manual or automated
        • The format for the tests resulted in quicker preparation time than traditional manual tests
        • Both developers and business testers could understand and use the same test format
        • Script maintenance was no longer a requirement
        • Tests could use any of the test execution tools required by the underlying technology
    • Business Analysts already using them, and use will grow
      • Home grown frameworks built within organisations to meet business demands
      • Niche suppliers providing frameworks; try a Google search – 512,000 hits this week
        • Seen a handful that appear mature
        • Latest review by Paul Herzlich (OVUM analyst)
      • Market Leaders such as Mercury developing Business Process Tester (BPT)
      This is the industry direction now
    • Automation Frameworks in Action
    • The project
      • Lloyds TSB Offshore in Channel Islands
      • Migration from current banking platform IBIS to Temenos Globus G13
      • Tight deadlines due to EU Savings Tax Directive coming into force on 1 st July 2005
      • Business processes required additional Globus modules
      • Multiple releases leading up to deadline required substantial testing
      • SQS-UK contacted to provide consultancy
    • The Team
      • SQS-UK recognised the requirement for test automation
      • Joint team from Odin Technology and SQS-UK assembled
      • Working alongside Lloyds TSB staff
      • Odin Technology provided testing technology
      • SQS-UK provided implementation resource and testing know-how
    • Project Approach
      • Split project roles for Automation Technicians and Testers
        • Three phase technical approach
          • Tool Evaluation & Technical Feasibility
          • Technical Implementation
          • Support
        • Test Analysis and Design throughout
      Tester Technician Tool Evaluation Technical Implementation Support Test Analysis & Design Project Time
    • Project Phase 1: Tool Evaluation and Technical Feasibility
    • Tool Evaluation
      • Automation isn’t always straightforward!
      • GUI Tools don’t work “out of the box” for all environments
      • They will almost certainly require some custom functionality to interact with the SUT
      • The Evaluation process:
        • Assess what custom functionality is required
        • Assess what involvement is required from an Automation Technician to provide the custom functionality
        • Assess if there are any other efficiency improvements that can be made through application of technology
      • This provides the scope of technical implementation phase
    • Project Phase 2: What would normally happen...
    • Typical Automation Project
      • Take existing Manual Scripts
      • Rework by hand into automated scripts & function libraries
      • The bulk of the work…
      • Building the Object Map
        • Providing the Mapping between the business terms used to describe the SUT Interface and the test tools requirements
      • Coding Test Scripts
        • Providing the instructions and associated data for the test tool to interact with the SUT and perform testing
    • Object Mapping
      • Providing the Mapping between the business terms used to describe the SUT Interface and the test tools’ requirements
      Application
    • Example Manual Tester Test Tool “ The Login Window” class: MSWDialog label: “Login.*”
    • Example 2 Manual Tester Test Tool “ Username” class: HTMLEdit id: Lgn_Uid
    • Object Mapping in a typical Automation Tool
      • Usually implemented under different names in different tools
        • GUI Map
        • Object Repository
        • Object Map
      • Mapping an Object
        • Start the Object recognition tool
        • Point at the Object with the Mouse Pointer
        • Select the Object
        • Give the Object a meaningful business name
        • Make manual amendments to the properties if required
    • Test Script Creation
      • Providing the instructions and associated data for the test tool to interact with the SUT and perform testing
      AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1
    • Scripting Approach
      • Use the Tools Scripting Language to construct business processes
      Function Login(ByVal UID As String, ByVal PWD As String) Dim rc rc = wait_window(“Login”,30) if rc <> 0 Then log_msg(“Window Not Found”, FALSE) Return End if focus_window(“Login”) click_on_text (“UserName”) type(UID & “{Return}”) click_on_text (“UserName”) type(PWD & “{Return}”) End Function
    • System Under Test Test Automation Tool AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 Reworking of Manual Scripts Application Object Maps
    • Project Phase 2: What was done differently… Taught an old dog some new tricks…
    • Test Design Model
      • A simple structured way of defining tests that is independent of the interface and the means of execution
    • Features of the Test Design Model
      • Easy to design tests without technical knowledge
      • Well structured
      • Can be used to describe tests for any interface
        • Graphical User Interfaces (GUI)
        • Non-UI Components (WebServices, APIs etc.)
      • Independent of Execution Mechanism
        • Manual testing
        • Automated testing
      • Not theory - proven 7 year history of implementations
      • In use in UK and Europe at many organisations for defining testing
    • Test Design Model Test Test steps A single test step Object Action Data Sub-tests
    • Test Model – GUI Example Test Login – enter user credentials Product search – enter product ID Product details – validate product info Logout Sub-test Login as user 1 Step Enter Username “jsmith”
    • A test step
      • Object + Action + Data
      • Action
        • SET enter data, navigation
        • GET retrieve data
        • VAL compare expected/actual data
    • How tests are designed
      • Using Microsoft Excel
      • Business processes are modelled
      • Sequences of steps and data are built up in Sub-Test Tables
      • Sequences of Sub-Tests model business processes
    • Interpreting the Model
      • There are products that can process the model
        • To generate Test Automation Scripts
          • For UI Testing using GUI Automation tools
          • For Non-UI Testing using Harnesses
        • To generate documentation for manual testing
    • Interpreting the Model Manual Scripts Non-UI Automated Scripts Automated Scripts
    • Globus A Self Descriptive Application
    • What is a Self-Descriptive Application?
      • An application that, through an automated process, can provide a description of its interfaces
      Application
    • The Globus User Interface How it works Globus Desktop Application Application A description of the interface for an application is stored in the database The Globus Desktop interprets the description and dynamically creates the user interface Windows and objects are presented to the user
    • How Globus Describes its Interface Custom Interpreter Globus Desktop Application API API can be invoked to provide the description without presenting it
    • A Sample of Globus Information APP: CUSTOMER_INST FIELD CUST_SNAME TYPE INPUT LABEL Surname LENGTH 20 SEQ 1 FIELD CUST_GEN TYPE OPTION LABEL Gender LENGTH 1 SEQ 7
    • How this Information was used
      • The Description gives us 3 things:
        • The business names used for the objects
        • The properties the test tool required for object identification
        • The sequence of the objects in the window
      • This could provide us with two things automatically:
        • The Object Map
        • The sequence of steps for our TDM sub-tests
    • More on Self-Descriptive Apps
      • Requirement on development to “name” objects in business terms
      • Microsoft .NET Applications
      • Windows Forms and Web Applications can provide us with
        • Object Maps
        • Sub-tests for the Test Design Model
      • This can drastically reduce the cost of automation
    • Overall - The Test System Design Axe Microsoft Excel Globus AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;).Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin &quot;6&quot; rc=AxeMainAPI.CheckDependency(&quot;5&quot;) If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin &quot;Home&quot; Browser(&quot;Browser&quot;). Basestate(AXEDIR & &quot;samplesOdinPortalhtmlindex.html&quot;) AxeMainAPI.BasestateEnd rc,&quot;&quot;,&quot;&quot; if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 Test Design Model & Object Maps Automation Scripts
    • Project Phase 2: Implementation
    • The Implementation
      • Technical Implementation
        • 9 days of Test Tool Technician
      • Using Automatic Interface Description in Globus
        • 1800+ Objects Mapped automatically
        • 90 Sub-test components generated
          • 30-75 steps per Sub-test
        • Time to Generate: 42 seconds
    • What was left for the Testers?
      • Test Design!
        • Designing scenarios for testing the application
        • Providing Test Data
        • Sequencing Sub-Tests to form business processes
    • Designing the Tests
      • 4 Testers – Using the Test Design Model
      • No previous automation experience
      • 40 Days Elapsed time
      • 1012 Complex Banking Scenarios Automated
    • Raw Statistics – Test Execution
      • Single Script
        • Automated = 80secs
        • Manual = 480secs
        • Automated is 6x Faster
      • Full Regression Pack
        • Automated = 22 Hrs (1 Workstation)
        • = 5.5 Hrs (4 Workstations)
        • Manual = 132 Hrs
        • = 22 Man/Days (using 6hrs productivity per day)
      • 22 Man/Days Testing in 5.5 Hrs.
    • Impact Upon You
    • Recommendations
      • Find out your organisations direction
      • Decide upon your own future
        • Increased management skills
        • Increased non-functional skills
        • Increased business skills
      • Ensure you understand how Automation Frameworks support testing so you have a view when asked, because you will be….!
    • Susan Windsor WMHL Consulting Limited, MD [email_address] www.wmhl.co.uk Closing slide Thank You Case Study provided by Odin Technology Ltd www.odin.co.uk