Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
CA - Take a value-driven approach to better software


Test Automation As A Service - Fusion (Sydney)

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Test Automation As A Service - Fusion (Sydney)

  1. 1. STANZ and Fusion (Friday @ 1:50pm, 14th September 2012) Test Automation as a Service Jonathon Wright (©
  2. 2. My starting point Twitter @Jonathon_Wright with how many slides in this presentation? First person to guess or be the closest! David Dubovsky, Boca Raton, 1999© BJSS Limited 2011 Commercial in Confidence 2
  3. 3. Bit of Background Three Decades in Automation Today 1998 2001 1994 --2001 2001 - 2005 2005 - 2008 2008 - 2011 Modularity driven testing Data driven testing Keyword Driven Hybrid Keyword/Data Driven 1994 - 1999 1998 1999 - 2001 2001 - 2004 2004 - 2005 2005 - 2006 2006 - 2008 2008 - 2010 2010 - 2011 WR x.x WinRunner 6.x WinRunner 7.x QTP 7.x QTP 8.x QTP 9.x QTP 10.x QTP 11 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
  4. 4. Automation is STILL a Silver Bullet? “….‘Silver Bullet’ refers to any straight forward solution perceived to have extreme effectiveness.” - Wikipedia
  5. 5. Automation is STILL prone to failure? Failure 32% 60% 40% Working 8% ROI 92% FAIL TO MEET TARGET ROI Cost of Failed Implementations 2004 2012 (Estimated) Industry: Test Automation (net worth) $1 billion $6 billion Automation Projects (failure cost) $½ billion $3 billion*Ovum study projects the testing tool market to be worth 56 billion US dollars by 2013 growing at an annual rate of 9.5%
  6. 6. Automation is STILL an expensive business? Manual (Prep + Execution) Manual (Pure Execution)Effort Manualmation Framework TAaaS Iterations “The cloud helps reduce ownership & maintenance costs, whilst moving away from expensive fixed-seat licensing models and un-utilised maintenance agreements.”
  7. 7. Automation is STILL time consuming “The implementation of a self-maintaining test asset loader/scraperthat automatically runs after each build and then self-validates againstthe universal testing language (DSL) and object models built within the cloud promote a more maintainable and portable solution.”
  8. 8. The current economic climate is making companies review their approach to IT even more closely “Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet accessallow for the development of approaches that allow companies to order testing as a service and pay only for what they use (PAYU).”
  9. 9. Moving into the Cloud “... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.” - Karl Flinders, Computer Weekly (2011)
  10. 10. Legacy Cloud (First Generation) Community Clouds SUT SUT SUT Private Public Combination TAaaS TAaaS Private Public Clouds Clouds “IBM are working with other cloud suppliers to establish standards for cloud service provider compliance.” - Murat Gunenc, IBM Cloud Representative @ Rhodes W1 (2011)
  11. 11. Testing in the Cloud A match made in heaven? and/or New Challenges & Opportunities ?
  12. 12. New Challenges Alternative Service Unique Services Provider? (Such as GPS) What happens when these services FAIL? Systemic Failure (DNS or Network) Sync
  13. 13. New Opportunities Community Clouds already tested?“Don’t expect the ecosystems (community or commercial) and other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)
  14. 14. Instant scalability Flexible Test Assets Flexible Test ExecutionEnvironments on-demand Actionable DefectsLive pause- playback Performance testing
  15. 15. Instant Accessibility 99.9% Availability (100% Web)
  16. 16. Solution Lifecycle Management• Any device• Anytime• Anywhere Instant Portability Best of Breed (BoB) tools on demand
  17. 17. No upfront investment Pay As You Use (PAYU) = Cost Management is easier Dont invest heavily upfront in something that may FAIL
  18. 18. Consumer Freedom “Enable sophisticated modelling of the components of clouddeployments, including servers, storage, databases and data. For the consumption of cloud resources” - (2012)
  19. 19. ShopLogin LoginChangePassword A0 LoginFrm A1 A2 Frm Client B1 B0 ShopHome (i.e.Tesco)TermsConditions MyShopHome B2 B3 EditProfileFrm Frm Frm 520 C0 CreateNewReport 7CreateNewReport C1 ReportType C2 ProductGroups C7 RunReport Frm Frm Frm Frm C12 ReportSche 603 487 duleFrm 97 C3 TimePeriods C8 Language 867 WorkStream Frm 122 67 32 (i.e. en-GB or en-US) 0 (i.e. UK, US or France) 139 C4 ProfileFrm C9 Index 789 588 D0 - Executed C5 Geographies C10 - Supported Frm D0 - Out of Scope 615 C6 ShopperPan elsFrm C11 0 Easier Process Integration CurrentReports CurrentReports Frm Application R0 F0 E0 CompletedReports (i.e. Shop) CompletedReports R1 OrderDetailsFrm Frm Flex Reports Excel S0 RecurringReports RecurringReports Buy.Tickets Frm B1 B4 B5 G0 B2 C2 B3 C3 MyGroups MyGroupFrm G1 ModifyGroupFrm A1 Login.Process C1 Check.Tickets Logout.Process C5 C4 D2 D1 D3 D4 D5 My.Account
  20. 20. Technology Process People Where are we going today?
  21. 21. Automation: Born in the cloud Application Lifecycle Management Unified Reporting Model Test Management Tool Team Foundation Server Reporting Dashboard Database Automation Automation Portal (TaaaS) Lab Management ObjectHandler.dll Windows Service EventHandler.dll DataHandler.dll Development (.NET 4.0) Application Programming Interface CodedUI Application Under Test 2003/2007/2010 “Pure ‘Testing as a Service’ Model - Born in the cloud (solution developed specifically with the cloud in mind).”
  22. 22. Taking the journey ‘to the cloud’ The Cloud (System Infrastructure) Customer Systems Intranet Desktop Site Application Web Site Web services Virtual lab Persistence Test Suite CMMI/ISO 9000 Software Build Rule Store Process Customer Automation Suite Automation Lifecycle Management (Test Automation as a Service Model) Business Level Test Asset Dashboard Customer Take on Configuration Customer supportKeyword Definition Management Flexible Cost Test Data Planning Monitoring/ Billing Development Operational management Support Credentials United Reporting Reporting Maintenance Deployment
  23. 23. How to: Consume cloud services End Users Test Automation as a Service (TaaaS) Run a test Cloud Web Endpoint «uses» TaaaS Portal «uses» Test Scheduler Service Schedule a future Silverlight test Dashboard «uses» QA Person Dashboard Service Results Service View real time test status «uses» Test Assets Repository (MTM / Quality Centre) Asset Scanner Service «uses» View historic test TD-API results Adapter Test Runner Environment - A Virtual Machine Business User created by the Framework VM Dispenser TestRunner Check Development Unified Reporting (Dashboard) Progress Framework Silverlight Technology Platform Test Data Results Test Adapter Dashboard Adapter Adapter Set Adapter Test Management View Live Defect Client System Test Repository Build Automation (Team City / TFS) Business Level Object Test Data Framework Results Build Client System Keywords Repository Sets Developer Object (BPM/BPT) Adapter
  24. 24. Technology: Test Designer Creating Cloud Tests any device, anytime and anywhere “Support for natural language with context sensitive validation.”© BJSS Limited 2011 (ATDD/BDD/TDD) Commercial in Confidence 24
  25. 25. Technology: Test Lab Designer Creating Cloud Environments featuring VM Dispenser Technology “Supports the full three tier application deployment.”© BJSS Limited 2011 Commercial in Confidence 25
  26. 26. Technology: Test Cloud Runners Executing Cloud Tests across a number of platforms “Monitor concurrent execution across a© BJSS Limited 2011 instantly scalable number of cloud clients.” Commercial in Confidence 26
  27. 27. Technology: Agnostic Solutions • The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which act as interpreters for the implementation agnostic language; Generate Select Select Select Test Run Environment Under Platform Test Technology Test (EUT) Adapter Adapter Adapter • Application Server • Windows 2012 • Load Testing • Coded-UI (Win32) • Database Server • Windows Hyper-V • Performance • Selenium ( • End-Client Machine • Windows 8 • Functional • Ranorex (Flex 4.0) “Platform, technology, client, browser, version, language agonistic solutions supporting cross browser testing© BJSS Limited 2011 through only a few clicks in the lab designer.” Commercial in Confidence 27
  28. 28. Technology: Event Driven Automation • Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change Passive ‒ Continuous build and integration or source code gated check-in Active© BJSS Limited 2011 Commercial in Confidence 28
  29. 29. Technology Process People Where are we going next?
  30. 30. Process: Automation Maturity Model Index (AMMi) Example Automation Projects Automation Level Investment in Effort Business Benefit Description AMMi Gen Implementation Planning Execution Accel. ROI Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112% Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85% Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26% Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200% Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186% TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306% “Sometimes half the battle is just starting companies on their Automation journey by taking them one step closer to becoming© BJSS Limited 2011 ready for automation.” Commercial in Confidence 30
  31. 31. Process: - Model Solution Lifecycle Management© BJSS Limited 2011 Commercial in Confidence 31
  32. 32. Process: Business Process Modelling v2.2“Business process modelling allows non subject matter experts to build and perform manual and automated tests without any previous domain expertise.”© BJSS Limited 2011 Commercial in Confidence 32
  33. 33. Process: Business Process Scenarios Request.VM B1 B4 B5 B2 C2 B3 C3 A1 Login.Process C1 Access.VM Logout.Process C5 C4 D2 D1 D3 D4 D5 Manage.VM Design complex tests scenarios from day one ‘first day’ testing (e.g. A1>B1>B3>C4>D3>C5) “Business Process Scenarios map dynamically to an instance of the© BJSS Limited 2011 Solution Under Test (SUT).” Commercial in Confidence 33
  34. 34. Process: Dynamic Test Assets Solution (e.g. Time to Market Improvement) Language Component / Application (e.g. en-GB or en-US) (e.g. Virtualisation Platform) Based on a moment in time Location (Geo) Version (e.g. UK, France, Asia) (e.g. Build, Release) Work Stream Client (e.g. Finance, Marketing) (e.g. Xerox, Thomson Reuters) “Test type agnostic – Pluggable test types so a smoke test, load test,cross browser test, concurrency test, penetration test can be utilised for multiple platforms. Not necessarily the same technology stack.”© BJSS Limited 2011 Commercial in Confidence 34
  35. 35. Process: Business Process Testing Action Object Event Domain Specific Language Login.Process Objects Actions Events Enter Username Text Enter Password Text Press Login Button Test Specific Language (TSL) Press | Login | Button “verbs and nouns in the test Verbs specific language are written in natural language which using Nouns context sensitive validation.”“The test editor is designed to be low waste, it will only offer nouns and verbs based on discovered artefacts not programming objects.”© BJSS Limited 2011 Commercial in Confidence 35
  36. 36. Process: Object Recognition (via Fuzzy Logic Matching) Test assets are defined by the self-maintaining test asset loader that uses fuzzy logic matching to identify UI objects 1.  XPATH Check 2.  Regular Expressions Please enter your password to logon 3.  Descriptive Programming Please enter your password to login Enter password here …… 4. Fuzzy Logic Enter password here …… logon logoff 5.n/a Artificial Intelligence login logoff“Fault tolerance is built in to report and continue on different levels of fuzzy matching. Combined with the non-technology specific test definition language, investment in ‘non-fragile’© BJSS Limited 2011 test assets is much more durable.” Commercial in Confidence 36
  37. 37. Process: Business Level Answers© BJSS Limited 2011 Commercial in Confidence 37
  38. 38. Businesses should focus on business,NOT be automation service providers
  39. 39. Ambassadors of Business Data Testing as a Service (TaaS) Dashboard (Realtime) Benefit Analysis Risk Dashboard Story Board (Presentation Layer) (Presentation Layer) (Presentation Layer) Business Stories Business Risk Business Domain Language Business Rules Project Risk Test Domain Language Business Workflow Technical Risk Business Questions Business Process Non-Functional Detailed Session Design (BPD) Requirements (NFR) Definition (DSD) (Visio Design Documentation) (Dynamic Data Source (XML)) (Dynamic Data Source (XAML)) Business Process Business Components Detailed Profile Modelling (BPM) & Interfaces Definition (DPD) (BPMNv2) (Dynamic Data Source (XAML)) (Dynamic Data Source (XAML)) Business Process Business Process Tests Detailed Test Scenarios (BPS) (BPT) Definition (DTD) (eXensible Process Definition Language) (Dynamic Data Source (XAML)) (Dynamic Data Source (XAML)) Business/Test Transactions Layer Dynamic Data Adapters Dynamic Test Adapters Solution Under Test (SUT) Layer “Integration into current Solution Lifecycle Management (SLM) to include real-time reporting dashboards powered by WCF based reporting frameworks.”© BJSS Limited 2011 Commercial in Confidence 39
  40. 40. Technology Process People The Final Frontier
  41. 41. People: Social Enterprise Ready Traditional TaaaS Documentation Living Meta Data Dashboards Funnel Virtualisation© BJSS Limited 2011 Commercial in Confidence 41
  42. 42. People: Social Enterprise Ready Activity Streams Real-time information Team Working Remote collaboration© BJSS Limited 2011 Commercial in Confidence 42
  43. 43. Who wants to do Automation?
  44. 44. People: Cross-functional teams Testers are testers, not programmers Anna the ScrumMaster Developers are developers, not testers“Many of the best tester’s are technically minded and can program but a tester’s real skill is applying testing knowledge to generate reusable test assets.”
  45. 45. People: Where are the testers? Time Zone 1 Time Zone 2 Time Zone 3 Community Tester - Platform: Hybrid Community Cloud (e.g. Facebook) Beta Tester - Platform: Public Cloud (e.g. Alpha Tester - Platform: Public/Private Cloud (e.g. Cloud Tester - Platform: Private Cloud (e.g. Internal Tester - Platform: Private Cloud (e.g. Testing Command Center (Domain Specific) • Supporting Core Testing Activities (Environment Monitoring/Management) •Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds) •Testing Management (Test Assets & Resource Enablement) Solution Command Center (Solution Specific) • End Client Dashboard/Reporting (KPIs) • Solution Management (Programme Level) •Service Provider Management (SLAs/Who does what when)
  46. 46. Automation should be a Global Marketplace “For many small-to-medium sized firms getting agent machines installed globally for performance test purposes is tricky / impossible. A highly scalable service to simply allow them to select a target location would add considerable value.”
  47. 47. Test Automation as a Service - Summary  Relevant – Clear traceability of the business value of Automation through the visualisation of the tests via Business Process Modelling (BPMNv2.2 compliant)  Effective – Self validating test assets achieved using natural language with context sensitive validation against business and testing rules, workflows and data  Maintainable - Self-maintaining test asset loader/scraper  Efficient - Reports on SUT health including ratings such as percentage availability since build/release, reported errors over time and traffic to error ratio  Manageable - Unified platform which non-domain experts can use a natural language to represent business processes and user story acceptance criteria  Portable - Platform, technology, client, browser, version, language agonistic  Reliable - Fault tolerance is built in to report and continue on different levels of fuzzy matching combined with the non-technology specific test definition language  Diagnosable - Actionable defects provided by environment under test (EUT) live pause-playback supported by dynamic data adapters (DDA) for accelerated defect investigation and resolution© BJSS Limited 2011 Commercial in Confidence 47 Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)
  48. 48. Testing Automation as a Service: Final Thoughts “The best aspects of these proven test approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution – just as the underpinning technology evolves over time.” My section in the ‘Experiences in Test Automation’ book (Dorothy Graham & Mark Fewster) 2012. Twitter with how many slides in this presentation?© BJSS Limited 2011 Commercial in Confidence 48 48
  49. 49. Further Information Hands on Demo ( User: michaeljackson Password: moonwalk Jonathon Lee Wright ( @jonathon_wright (twitter) | (blog) |
  • sky_wu

    Oct. 18, 2014

    Sep. 28, 2013
  • lsalazar2002

    Sep. 27, 2012


Total views


On Slideshare


From embeds


Number of embeds