Test Automation: A Roadmap For Sucesss
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Test Automation: A Roadmap For Sucesss

  • 4,086 views
Uploaded on

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,086
On Slideshare
4,081
From Embeds
5
Number of Embeds
1

Actions

Shares
Downloads
139
Comments
0
Likes
1

Embeds 5

http://softtest.ie 5

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 2008-01-20 Test Automation A Roadmap For Success Scott Armstrong Belfast, February 2010 SQS Group Limited Agenda Steps for Success Automation Tool Selection Test Selection and Prioritisation Implementing the Strategy SQS Case Studies Questions © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 2© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 1
  • 2. 2008-01-20 Steps for Success Approaching Test Automation Planning the Framework © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Approaching Test Automation Steps for Success DESIGN Build maintainable tests Target and prioritise test areas PLAN TOOL Treat as a Select with care Development Project Pilot tool Manage Expectations Manage Commitment at all Implementation levels AUTOMATION PROJECT TEAM TEST MANAGEMENT Code reuse and Defect Management collaboration Config Management Partnership between Information and test and dev Reporting Requirements © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 2
  • 3. 2008-01-20 Planning the Framework Steps for Success What are the Management Information Requirements Contents Format What are the Execution Requirements? On-demand Automated Nightly Builds What technologies are we testing? Embedded Operating System What resources are available? Hardware Technical Expertise Are there cultural or organisational considerations? Outsourced IT Management – no admin access Current partnership with tool vendor © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Automation Tool Selection Selection Process Identify Requirements Buy, Open Source or Build © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 3
  • 4. 2008-01-20 Selection Process Automation Tool Selection Define Problem Shortlist Consider Define Evaluate automation required Decide as a solution Demo features Make Trial business case Constraints © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Identify Requirements Automation Tool Selection Start by evaluating your requirements, not by investigating the tool market. Look at the current problems – what has prompted the search for an automated solution: Manual testing problems (e.g. too time consuming, error prone); No time for regression testing when small changes are made to the software; Set-up of test data or test cases is error prone; Inadequate test documentation; Don’t know how much of the software has been tested; Testing is ineffective. Not all of these problems are best solved by automation. Rank your problems by importance or current cost to the organisation. © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 4
  • 5. 2008-01-20 Buy, Open Source or Build Automation Tool Selection • Quickly provides a high level of features Commercial • Well supported in terms of documentation, help and training • Often an attractive project to be involved with Tool • Added cost of license fees • No license fee cost Open Source • Limited to support from community forums • Defects in tool may not be addressed Tool • Tool may lack a clear roadmap; may not be future-proof • No license fee cost Build • Development and maintenance can prove expensive • Designed to meet project requirements Bespoke Tool • Lacks full range of features © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Test Selection and Prioritisation Test Selection Test Prioritisation © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 5
  • 6. 2008-01-20 Test Selection Test Selection and Prioritisation Some tests cannot be automated. Some tests are not worth automating. May take more time to automate tests than to run them manually You may not have to automate everything to get significant benefits 10% of the tests may be run 90% of the time. Some types of tests are amenable to automation, some not: Functional tests are often the most straightforward Performance tests are notoriously difficult to carry out manually. These are a good candidate for automation. Non-functional tests can cover maintainability, portablility, testability, usability, etc, and can be inappropriate for automation. © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Test Prioritisation Test Selection and Prioritisation Do not automate too much too soon. Best practices take time to develop. Go for ‘quick wins’ – the areas where automation will have the largest impact most quickly. Prioritisation factors: A set of breadth tests (sample each system area); Tests for the most important functions; Tests that are easiest to automate; Tests that will give the quickest payback; Tests that are run the most often. © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 6
  • 7. 2008-01-20 Implementing the Strategy Automation Team Fit Maintenance Model © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Automation Team Fit Implementing the Strategy Specialist Group/Tool Support Subject Test Team Matter Expert Automation Team Development Management Team Buy-in © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 7
  • 8. 2008-01-20 Maintenance Models Implementing the Strategy Knowledge • Client representative with technical background is trained during automation project • Maintenance of current scripts can be supported Transfer independently for future product releases • Future new tests can be automated in-house and added to the regression pack Managed • Client does not have to be involved in the details of automation development, execution or maintenance • The necessary technical skills do not need to exist in- Service house • Assured best practice automation and quality © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 SQS Case Studies Lloyds TSB Wholesale Banking Local Utility Company © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 8
  • 9. 2008-01-20 Lloyds TSB Wholesale Banking SQS Case Studies Manual Testing Challenges Our Engagement • 600+ Tests and growing • Innovative test automation solution implementation • 200 man days per release • Highly maintainable and extensible solution • Concerns over coverage • Successful solution handover to client test team • Regression pack growth impacted by limited resources Lloyds TSB Customer Success Return on Investment • Increased test execution efficiency by 85% “Without “The automated • Reduction in deployed defects >50% automation, we regression suite • Enabled daily smoke test execution would not have made my job • Overall ROI of £1.9m over 6 years gone live” more interesting” © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 Local Utility Company SQS Case Studies Manual Testing Challenges Our Engagement • 100 test scripts with multiple complex data-driven • Collaborative design of framework specific to client iterations requirements • 120 man days regression testing per year • Full knowledge transfer to existing test team for • Low test coverage – limited to ‘spot checks’ ongoing maintenance and development • Testing of new functionality impacted by large • Modular framework allows new tests to be built with regression packs minimal effort Local Utility Customer Success Return on Investment • SQS are now engaged on further automation • Saving of 96 man days of execution per year projects with the parent company • Data-driven scripts increased test coverage by a • The existing test team continue to expand and factor of approximately 200 maintain the framework to increase their ROI • Automation solution was ported to a further 2 business units, multiplying the ROI © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 9
  • 10. 2008-01-20 SQS Group Limited The Concourse | Northern Ireland Science Park Queen‘s Road | Queen‘s Island | Belfast BT3 9DT Phone: +44 (0) 2890 022 660 E-Mail: scott.armstrong@sqs-uk.com Internet: www.sqs-uk.com Thank you for your attention © SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010© SQS Group Limited | Presentationtitle | July 2008 | page ‹Nr.› 10