1.
STARWest (Thursday @ 1:30 PM, 4th October 2012) Test Automation as a Service Jonathon Wright (jlwright@iee.org)© TestingAsAService.com www.linkedin.com/in/automation

2.
My starting point Twitter @Jonathon_Wright with how many slides in this presentation? First person to guess or be the closest! GOTTAWANNAWIN Quote - David Dubovsky, Boca Raton, 1999© BJSS Limited 2011 Commercial in Confidence 2

3.
The current economic climate is making companies review their approach to IT even more closely • Why (Benefits) • What (Cloud Automation) • When (Timeline) • Scope (Features) • Transition (AMMi) • Dependencies (Cloud Type) • Implementation (Agnostic) • Universal Test Language • Maintenance (Asset Loaders) • Cloud Virtualisation • Continuous Build & Integration • Event Driven Automation  Dashboards (Real-time)  Funnel Virtualisation “Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet accessallow for the development of approaches that allow companies to order testing as a service and pay only for what they use (PAYU).”

4.
Why? – Bit of Background Three Decades in Automation Today 1998 2001 1994 --2001 2001 - 2005 2005 - 2008 2008 - 2011 Modularity driven testing Data driven testing Keyword Driven Hybrid Keyword/Data Driven 1994 - 1999 1998 1999 - 2001 2001 - 2004 2004 - 2005 2005 - 2006 2006 - 2008 2008 - 2010 2010 - 2011 WR x.x WinRunner 6.x WinRunner 7.x QTP 7.x QTP 8.x QTP 9.x QTP 10.x QTP 11 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

5.
No longer one Tool to rule them all? Recent Explosion in Automation Tools

6.
Automation is STILL prone to failure? Failure 32% 60% 40% Working 8% ROI 92% FAIL TO MEET TARGET ROI Cost of Failed Implementations 2004 2012 (Estimated) Industry: Test Automation (net worth) $1 billion $6 billion Automation Projects (failure cost) $½ billion $3 billion*Ovum study projects the testing tool market to be worth 56 billion US dollars by 2013 growing at an annual rate of 9.5%

7.
Automation is STILL an expensive business? Manual (Prep + Execution) Manual (Pure Execution)Effort Manualmation Framework TAaaS Iterations “The implementation of a self-maintaining test asset loader/scraper that automatically runs after each build and then self-validates against the universal testing language (DSL) and object models built within the cloud promote a more maintainable and portable solution.”

8.
Introducing Legacy Cloud (First Generation) Community Clouds SUT SUT SUT Private Public Combination TAaaS TAaaS Private Public Clouds Clouds “... the prediction that 60% ofcloud suppliers to establish “IBM are working with other the average enterprise will the standards for cloud service provider compliance.” have 60% of its applications in the cloud.” - Murat Gunenc, Flinders, Computer WeeklyRhodes W1 (2011) - Karl IBM Cloud Representative @ (2011)

9.
Testing in the Cloud A match made in heaven? and/or New Challenges & Opportunities ?

10.
New Challenges Alternative Service Unique Services Provider? (Such as GPS) What happens when these services FAIL? Systemic Failure (DNS or Network) Sync

11.
New Opportunities Community Clouds already tested?“Don’t assume the ecosystems (community or commercial) and other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)

12.
Instant scalability Instant Accessibility Flexible Test Assets Environments on-demandFlexible Test Execution Consumer Freedom Geo-based execution 99.9% Availability (100% Web) CompareTesting.com Instant Portability No upfront investment “Enable sophisticated modelling of the components of clouddeployments, including servers, storage, databases and data.Best of Breed consumption of cloud resources” - Pay As You Use (PAYU) For the (BoB) tools on demand PlanForCloud.com (2012)

13.
Technology Process People Where are we going today?

14.
Automation: Born in the cloud Application Lifecycle Management Unified Reporting Model Test Management Tool Team Foundation Server Reporting Dashboard Database Automation Automation Portal (TaaaS) Lab Management ObjectHandler.dll Windows Service EventHandler.dll DataHandler.dll Development (.NET 4.0) Application Programming Interface CodedUI Application Under Test 2003/2007/2010 “Pure ‘Testing as a Service’ Model - Born in the cloud (solution developed specifically with the cloud in mind).”

15.
Taking the journey ‘to the cloud’ The Cloud (System Infrastructure) Customer Systems Intranet Desktop Site Application Web Site Web services Virtual lab Persistence Test Suite CMMI/ISO 9000 Software Build Rule Store Process Customer Automation Suite Automation Lifecycle Management (Test Automation as a Service Model) Business Level Test Asset Dashboard Customer Take on Configuration Customer supportKeyword Definition Management Flexible Cost Test Data Planning Monitoring/ Billing Development Operational management Support Credentials United Reporting Reporting Maintenance Deployment

16.
How to: Consume cloud services End Users Test Automation as a Service (TaaaS) Run a test Cloud Web Endpoint «uses» TaaaS Portal «uses» Test Scheduler Service Schedule a future Silverlight test Dashboard «uses» QA Person Dashboard Service Results Service View real time test status «uses» Test Assets Repository (MTM / Quality Centre) Asset Scanner Service «uses» View historic test TD-API results Adapter Test Runner Environment - A Virtual Machine Business User created by the Framework VM Dispenser TestRunner Check Development Unified Reporting (Dashboard) Progress Framework Silverlight Technology Platform Test Data Results Test Adapter Dashboard Adapter Adapter Set Adapter Test Management View Live Defect Client System Test Repository Build Automation (Team City / TFS) Business Level Object Test Data Framework Results Build Client System Keywords Repository Sets Developer Object (BPM/BPT) Adapter

17.
Technology: Test Designer Creating Cloud Tests any device, anytime and anywhere “Supporting natural language with context sensitive validation.”© BJSS Limited 2011 (ATDD, BDD & TDD Compatible) Commercial in Confidence 17

18.
Technology: Test Lab Designer Creating Cloud Environments featuring VM Dispenser Technology “Supports the full three tier application deployment.”© BJSS Limited 2011 Commercial in Confidence 18

19.
Technology: Test Cloud Runners Executing Cloud Tests across a number of platforms*iPhone5 (iOS6.0 not yet fully tested) “Monitor concurrent execution across a© BJSS Limited 2011 instantly scalable number of cloud clients.” Commercial in Confidence 19

20.
Technology: Agnostic Solutions • The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which act as interpreters for the implementation agnostic language; Generate Select Select Select Test Run Environment Under Platform Test Technology Test (EUT) Adapter Adapter Adapter • Application Server • Windows 2012 • Performance • Coded-UI (Win32) • WebServices • Windows 8 • Load • Selenium (ASP.net) • Database Server • Windows Hyper-V • Functional • Ranorex (Flex 3/4) • Schema • Windows 7 • Component • 1.2.0 (Flex 2) • Data • Windows 2008 R2 • Integration • 2.3.3 (Flex 3) • End-Client Machine • Unit Level • 3.0.2 (Flex 4) • UI Level • Automation.UI (WPF) • White (SWT(Java)) “Platform, technology, client, browser, version, language agonistic solutions supporting cross browser testing© BJSS Limited 2011 through only a few clicks in the lab designer.” Commercial in Confidence 20

21.
Technology: Event Driven Automation • Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change Passive ‒ Continuous build and integration or source code gated check-in Active© BJSS Limited 2011 Commercial in Confidence 21

22.
Technology Process People Where are we going next?

23.
Process: Automation Maturity Model Index (AMMi) Example Automation Projects Automation Level Investment in Effort Business Benefit Description AMMi Gen Implementation Planning Execution Accel. ROI Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112% Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85% Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26% Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200% Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186% TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306% “Sometimes half the battle is just starting companies on their automation journey by taking them one step closer to becoming© BJSS Limited 2011 ready for automation.” Commercial in Confidence 23

24.
Process: TestingAsAService.com - Model Solution Lifecycle Management TEST AUTOMATION AS A SERVICE© BJSS Limited 2011 Commercial in Confidence 24

25.
Process: Business Process Modelling v2.2“Business process modelling allows non-subject matter experts (SMEs) to build and perform manual and automated tests without any previous domain expertise.”© BJSS Limited 2011 Commercial in Confidence 25

26.
Process: Business Process Scenarios B1 Request.VM Request.VM A1 C3 C4 C4 C2 C3 C1 D1 C1 D2 D1 C2 D2 Login.Process Login.Process A1 A2 Accept.Terms Access.VM B2 Access.VM Logout.Process Logout.Process D3 D3 D4 E1 D4 E1 A3 E2 E2 E3 E3 E4 E4 B3 Manage.VM Manage.VM Design complex tests scenarios from day one ‘first day’ testing (Release 1.0 = A1>C3>E1>D4>E3>C4) (Release 2.0 = A1>B1>C3>E1>D4>E3>C4) “Business Process Scenarios map dynamically to an instance of the© BJSS Limited 2011 Solution Under Test (SUT).” Commercial in Confidence 26

27.
Process: Dynamic Test Assets Solution (e.g. Time to Market (against new builds via asset loader) Improvement) Application / (against moment in time) Language (e.g. en-GB Component (e.g. Virtualisation Validate tests or en-US) Validate tests Platform) Emerging Test Assets Location (Geo) Version (e.g. UK, (e.g. Build, France, Asia) Release) Work Stream Client (e.g. Finance, (e.g. Xerox, Marketing) Thomson Reuters) “Test type agnostic – Pluggable test types so a smoke, regression,integration & performance can be reused against multiple platforms.”© BJSS Limited 2011 Commercial in Confidence 27

28.
Process: Object Recognition (via Fuzzy Logic Matching) Test assets are defined by the self-maintaining test asset loader that uses fuzzy logic matching to identify UI objects 1.  XPATH Check 2.  Regular Expressions 3.  Descriptive Programming Please enter your password to logon 4. Fuzzy Logic Please enter your password to login Enter password here …… Enter password here …… logon logoff 5.n/a Artificial Intelligence login logoff 6. Social Intelligence“Fault tolerance is built in to report and continue on different levels of fuzzy matching. Combined with the non-technology specific test definition language, investment in ‘non-fragile’© BJSS Limited 2011 test assets is much more durable.” Commercial in Confidence 28

29.
Next Generations Collaborating Together in Community Testing Cloud? Businesses should focus on business, NOT building automation frameworksGen Zero 1st Gen 2nd Gen 3rd Gen 4th Gen 5st Gen(Record) (Modular) (Data) (Keyword) (Hybrid) ????(Scriptless)????

30.
Actionable Insight (Ambassadors of Data) Testing as a Service (TaaS) Dashboard (Realtime) Benefit Analysis Risk Dashboard Story Board (Presentation Layer) (Presentation Layer) (Presentation Layer) Business Stories Business Risk Business Domain Language Business Rules Project Risk Test Domain Language Insight Business Workflow Technical Risk Business Questions Business Process Non-Functional Detailed Session Design (BPD) (Visio Design Documentation) Funnel Virtualisation Requirements (NFR) (Dynamic Data Source (XML)) Definition (DSD) (Dynamic Data Source (XAML)) Business Process Business Components Detailed Profile Modelling (BPM) & Interfaces Definition (DPD) (BPMNv2) (Dynamic Data Source (XAML)) (Dynamic Data Source (XAML)) Business Process Business Process Tests Detailed Test Scenarios (BPS) (BPT) Definition (DTD) Data Data(Dynamic DataData Data Data (Dynamic Data Source (XAML)) (eXensible Process Definition Language) Data Source (XAML)) Data DataBusiness/Test Data Data Data Data Data Transactions Layer Data Data Data Data Data Dynamic Data Adapters Dynamic Test Adapters Solution Under Test (SUT) Layer “Integration into current Solution Lifecycle Management (SLM) to include real-time reporting dashboards poweredConfidence based reporting frameworks.” 30© BJSS Limited 2011 Commercial in by WCF

31.
Technology Process People The Final Frontier

32.
People: Cross-functional teams Testers are testers, not programmers Julie the ScrumMaster Developers are developers, not testers“Many of the best tester’s are technically minded and can program but a tester’s real skill is applying testing knowledge to generate reusable test assets.”

33.
People: Social Enterprise Ready Traditional TaaS Documentation Living Meta Data “Master the Data” Inditex Dashboards Funnel Virtualisation “Encourage simplicity” D’O© BJSS Limited 2011 Commercial in Confidence 33

34.
People: Social Enterprise Ready Activity Streams Real-time information “Embrace cloud workforces” Mosaic Team Working Remote collaboration “Collaborate inventively” McLaren© BJSS Limited 2011 Commercial in Confidence 34

35.
People: Where are the testers? Time Zone 1 Time Zone 2 Time Zone 3 Community Tester - Platform: Hybrid Community Cloud (e.g. Facebook) Beta Tester - Platform: Public Cloud (e.g. beta.bbc.co.uk) Alpha Tester - Platform: Public/Private Cloud (e.g. alpha.bbc.co.uk) Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.bbc.co.uk) Internal Tester - Platform: Private Cloud (e.g. internal.bbc.co.uk) Testing Command Center (Domain Specific) • Supporting Core Testing Activities (Environment Monitoring/Management) •Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds) •Testing Management (Test Assets & Resource Enablement) Solution Command Center (Solution Specific) • End Client Dashboard/Reporting (KPIs) • Solution Management (Programme Level) •Service Provider Management (SLAs/Who does what when)

36.
Automation should be a Global Marketplace Smaller, possibly remote firms or individuals can now have easy PAYU access to world class expertise.

37.
Test Automation as a Service - Summary Clear traceability of the business value of Automation through the visualisation of Relevant the tests via Business Process Modelling (BPMNv2.2 compliant) Self-validating test assets achieved using natural language with context sensitive Effective validation against business and testing rules, workflows and data Maintainable Self-maintaining test asset loader/scraper Reports on SUT health including ratings such as percentage availability since Efficient build/release, reported errors over time and traffic to error ratio Unified platform which non-domain experts can use a natural language to Manageable represent business processes and user story acceptance criteria Technology agonistic - Platform, client/component, browser, version & language Portable Test type agnostic – smoke, regression, integration & performance Fault tolerance is built in to report and continue on different levels of fuzzy Reliable matching combined with the non-technology specific test definition language Actionable defects provided by environment under test (EUT) live pause-playback Diagnosable supported by dynamic data adapters (DDA) for accelerated defect investigation and resolution© BJSS Limited 2011 Commercial in Confidence 37 Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)

38.
Testing Automation as a Service: Final Thoughts “The best aspects of these proven test approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution – just as the underpinning technology evolves over time.” My section in the ‘Experiences in Test Automation’ book (Dorothy Graham & Mark Fewster) 2012.© BJSS Limited 2011 Commercial in Confidence 38

39.
Further Information Hands on Demo (Hyper-Test.com) User: michaeljackson Password: moonwalk Jonathon Lee Wright (jonathon.wright@testing.uk.com) @jonathon_wright (twitter) | www.TaaaS.net (blog) www.automation.org.uk | automation.sharepoint.com www.linkedin.com/in/automation