We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice? Is it really the perfect environment to create powerful testing solutions? If so, why would your businesses invest in creating its own custom test automation frameworks to support your solution lifecycle management, when there’s a solution to meet everyone’s needs? Based on his experience, describing recent successes and the pitfalls to be avoided, Jonathon Wright explores these questions, and the pros and cons of using the “Testing as a Service” model. Sharing real world examples, he shows how you can transform the automation services in your organization and how you can take your company on the journey into the cloud regardless of your automation maturity. Influence and contribute to the future direction in the “Testing as a Service” community
The current economic climate is making companies review their approach to IT even more closely • Why (Benefits) • What (Cloud Automation) • When (Timeline) • Scope (Features) • Transition (AMMi) • Dependencies (Cloud Type) • Implementation (Agnostic) • Universal Test Language • Maintenance (Asset Loaders) • Cloud Virtualisation • Continuous Build & Integration • Event Driven Automation Dashboards (Real-time) Funnel Virtualisation “Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet accessallow for the development of approaches that allow companies to order testing as a service and pay only for what they use (PAYU).”
No longer one Tool to rule them all? Recent Explosion in Automation Tools
Automation is STILL prone to failure? Failure 32% 60% 40% Working 8% ROI 92% FAIL TO MEET TARGET ROI Cost of Failed Implementations 2004 2012 (Estimated) Industry: Test Automation (net worth) $1 billion $6 billion Automation Projects (failure cost) $½ billion $3 billion*Ovum study projects the testing tool market to be worth 56 billion US dollars by 2013 growing at an annual rate of 9.5%
Automation is STILL an expensive business? Manual (Prep + Execution) Manual (Pure Execution)Effort Manualmation Framework TAaaS Iterations “The implementation of a self-maintaining test asset loader/scraper that automatically runs after each build and then self-validates against the universal testing language (DSL) and object models built within the cloud promote a more maintainable and portable solution.”
Introducing Legacy Cloud (First Generation) Community Clouds SUT SUT SUT Private Public Combination TAaaS TAaaS Private Public Clouds Clouds “... the prediction that 60% ofcloud suppliers to establish “IBM are working with other the average enterprise will the standards for cloud service provider compliance.” have 60% of its applications in the cloud.” - Murat Gunenc, Flinders, Computer WeeklyRhodes W1 (2011) - Karl IBM Cloud Representative @ (2011)
Testing in the Cloud A match made in heaven? and/or New Challenges & Opportunities ?
New Challenges Alternative Service Unique Services Provider? (Such as GPS) What happens when these services FAIL? Systemic Failure (DNS or Network) Sync
New Opportunities Community Clouds already tested?“Don’t assume the ecosystems (community or commercial) and other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)
Instant scalability Instant Accessibility Flexible Test Assets Environments on-demandFlexible Test Execution Consumer Freedom Geo-based execution 99.9% Availability (100% Web) CompareTesting.com Instant Portability No upfront investment “Enable sophisticated modelling of the components of clouddeployments, including servers, storage, databases and data.Best of Breed consumption of cloud resources” - Pay As You Use (PAYU) For the (BoB) tools on demand PlanForCloud.com (2012)
Technology Process People Where are we going today?
Automation: Born in the cloud Application Lifecycle Management Unified Reporting Model Test Management Tool Team Foundation Server Reporting Dashboard Database Automation Automation Portal (TaaaS) Lab Management ObjectHandler.dll Windows Service EventHandler.dll DataHandler.dll Development (.NET 4.0) Application Programming Interface CodedUI Application Under Test 2003/2007/2010 “Pure ‘Testing as a Service’ Model - Born in the cloud (solution developed specifically with the cloud in mind).”
Taking the journey ‘to the cloud’ The Cloud (System Infrastructure) Customer Systems Intranet Desktop Site Application Web Site Web services Virtual lab Persistence Test Suite CMMI/ISO 9000 Software Build Rule Store Process Customer Automation Suite Automation Lifecycle Management (Test Automation as a Service Model) Business Level Test Asset Dashboard Customer Take on Configuration Customer supportKeyword Definition Management Flexible Cost Test Data Planning Monitoring/ Billing Development Operational management Support Credentials United Reporting Reporting Maintenance Deployment
How to: Consume cloud services End Users Test Automation as a Service (TaaaS) Run a test Cloud Web Endpoint «uses» TaaaS Portal «uses» Test Scheduler Service Schedule a future Silverlight test Dashboard «uses» QA Person Dashboard Service Results Service View real time test status «uses» Test Assets Repository (MTM / Quality Centre) Asset Scanner Service «uses» View historic test TD-API results Adapter Test Runner Environment - A Virtual Machine Business User created by the Framework VM Dispenser TestRunner Check Development Unified Reporting (Dashboard) Progress Framework Silverlight Technology Platform Test Data Results Test Adapter Dashboard Adapter Adapter Set Adapter Test Management View Live Defect Client System Test Repository Build Automation (Team City / TFS) Business Level Object Test Data Framework Results Build Client System Keywords Repository Sets Developer Object (BPM/BPT) Adapter
Next Generations Collaborating Together in Community Testing Cloud? Businesses should focus on business, NOT building automation frameworksGen Zero 1st Gen 2nd Gen 3rd Gen 4th Gen 5st Gen(Record) (Modular) (Data) (Keyword) (Hybrid) ????(Scriptless)????
People: Cross-functional teams Testers are testers, not programmers Julie the ScrumMaster Developers are developers, not testers“Many of the best tester’s are technically minded and can program but a tester’s real skill is applying testing knowledge to generate reusable test assets.”