BCS - SIGIST - Test Automation in the Hybrid Cloud

8,910 views

Published on

What is the future of test automation? The possibilities associated with hybrid cloud computing provide instant
scalability, flexibility and availability for testing on demand with no upfront investment. This provides the
industry with a perfect opportunity to utilise powerful test automation platform solutions.

The global testing cloud marketplace will allow for the joint collaboration of leading test specialists
following industry best practice. This enables firms of
all sizes to access the latest test approaches and methodologies whilst providing a unified platform for
domain experts to represent business processes and user story acceptance criteria in a natural
language.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
8,910
On SlideShare
0
From Embeds
0
Number of Embeds
6,972
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BCS - SIGIST - Test Automation in the Hybrid Cloud

  1. 1. BCS SIGIST Winter 2013 Conference (Thursday @ 2:45 PM, 5th December 2013) Test Automation in the (Hybrid) Cloud @jonathon_wright #BCS_SIGIST Jonathon Wright (jonathon.wright@db.com) http://www.linkedin.com/in/automation
  2. 2. My starting point: Founder of the (Hybrid) Approach Pattern (2003) @jonathon_wright #BCS_SIGIST www.sigist.org.uk
  3. 3. The current economic climate is making companies review their approach to IT even more closely • Why (Business Benefits & Value) • What (Cloud Test Automation) • When (Timeline) • Agile Portfolio Management (Scope) • Transition (AMMi) • Dependencies (Cloud Type) • Design (First Day Testing) • Design (Manual’mation) • Implementation (Agnostic) • Cloud Platform (IaaS) • High Volume Automation (HVAT) • Continuous Build & Delivery • • Dashboards (Real-time) Actionable Insight (Funnel Virtualisation) “We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant @jonathon_wright scalability, flexibility, and availability for testing-on-demand. - But how well does this work in practice?” #BCS_SIGIST
  4. 4. Background – The Journey to the Cloud TEST AUTOMATION AS A SERVICE (TAaaS) REAL-TIME VISIBILITY AND CONTROL INSTANT SCALABILITY SCALE UP AND DOWN ISO 29119 SLA’s AMMI PAY AS YOU USE TESTING AS A SERVICE (TaaS) PERFORMANCE TESTING AS A SERVICE (PAYU) INFRASTRUCTURE AS A SERVICE (IaaS) (PTaaS) HIGH AVAILABILITY PLATFORM AS A SERVICE SOFTWARE AS A SERVICE (PaaS) (SaaS)
  5. 5. Background – Three Decades in Automation – Mid-Late 90s SUT AUT Gen Zero (Linear) License x1 Budget in pounds, company in thousands @jonathon_wright #BCS_SIGIST Product Test Engineer Tool(s): 1 Teams:5 Locations:2 Mainframe
  6. 6. SUT Background – Three Decades in Automation – Late 90s AUT 1st Gen (Modular) Automation Framework Architect Core Hours 1am to 10am (EST) Graduate Testers Core Hours 9am to 5pm (EST) Automation Engineers Core Hours 9am to 5pm (GMT) Licenses 10+ Budget in thousands, company in millions http://www.google.co.in/patents/US7526498 @jonathon_wright #BCS_SIGIST Product Test Engineer Tool(s): 3 Teams:10 Locations:5 Client Server
  7. 7. SUT Background – Three Decades in Automation – Early 00s AUT 2nd Gen (Data) Automation Consultants (UK) Core Hours 9am to 5pm (GMT) Development (2 week sprints) Core Hours 9am to 5pm (GMT+1) Licenses x25 Budget in hundreds of thousands @jonathon_wright #BCS_SIGIST Automation Consultant Tool(s): 5 Teams:15 Locations:5 Client Server
  8. 8. Background – Three Decades in Automation – Late 00s Automation Consultants (NY) Core Hours 9am to 5pm (EST) SUT AUT Automation Consultants (UK) Core Hours 9am to 5pm (GMT) 3rd Gen (Keyword) Automation Consultants 9am to 5pm (UTC+8) Licenses 100+ Development Capability (US) Core Hours 9am to 5pm Development Capability (EUR) Core Hours 9am to 5pm Budget in millions, company in billions @jonathon_wright #BCS_SIGIST Automation Framework Architect Tool(s): 15+ Teams:100+ Locations:20+ Web
  9. 9. Background – Three Decades in Automation – Early 10s Automation Consultants (NY) Core Hours 9am to 5pm (EST) SUT AUT Automation Consultants (UK) Core Hours 9am to 5pm (GMT) 4th Gen (Hybrid) Automation Consultants 9am to 5pm (UTC+8) Licenses PAYU Development Capability (EUR) Core Hours 9am to 5pm Budget in millions, company in billions @jonathon_wright #BCS_SIGIST Global Head of Automation Tool(s): 20+ Teams:50+ Locations:25+ Cloud
  10. 10. Automation Consultants (NY) Core Hours 9am to 5pm (EST) wAgile Teams (US) Core Hours 9am to 5pm (EST) SUT Automation Consultants (UK) Core Hours 9am to 5pm (GMT) AUT (TaaaS) Automation Consultants 9am to 5pm (UTC+8) wAgile Teams (Asia) Core Hours 9am to 5pm (UTC+8) Background – Three Decades in Automation – Today 5th Gen wAgile Teams (EUR) Core Hours 9am to 5pm (CET) Budget 78 million, company investment 4 billion @jonathon_wright #BCS_SIGIST Agile Portfolio Management Tool(s): 50+ Teams:1k+ Locations:100s Cloud
  11. 11. Background: Where's testers? – BCS SIGIST Winter 2011 Conference Time Zone 1 Time Zone 2 Time Zone 3 Community Tester - Platform: Hybrid Community Cloud (e.g. MyDb.com) Beta Tester - Platform: Public Cloud (e.g. sdod.db.com) Alpha Tester - Platform: Public/Private Cloud (e.g. dbras.db.com) Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.db.com) Internal Tester - Platform: Private Cloud (e.g. dod.internal.db.com) Testing Command Center (Domain Specific) • Supporting Core Testing Activities (Environment Monitoring/Management) •Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds) •Testing Management (Test Assets & Resource Enablement) Solution Command Center (Solution Specific) • End Client Dashboard/Reporting (KPIs) • Solution Management (Programme Level) •Service Provider Management (SLAs/Who does what when)
  12. 12. Background – Testing as a Service – BCS SIGIST Winter 2011 Conference “Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet access allow for the development of approaches that allow companies to order testing @jonathon_wright as a service and pay only for what they use.” #BCS_SIGIST
  13. 13. Background – Test Automation as a Service www.TaaaS.net “Over the three years we have been developing a azure cloud based solution to provide true "Test Automaton as a Service" using a hybrid automation approach specifically designed with the Hybrid @jonathon_wright Cloud (Private/Public) in mind to utilise the instant scalability, flexibility and availability of the Cloud.” #BCS_SIGIST
  14. 14. Background – Test Automation as a Service www.slideshare.net/Jonathon_Wright “The current implementation is a service-oriented architecture (SOA) based on utilising business process modelling to create business level keywords written in natural language (domain specific languages) validated against content sensitive validation to create reusable, jigsaw pieces representing the Solution @jonathon_wright www.sigist.org.uk Under Test (SUT).” #BCS_SIGIST
  15. 15. Background – Test Automation as a Service Creating Cloud Tests natural language validated against content sensitive validation Executing Cloud Tests platform, technology, client, browser, version, language, test type agnostic @jonathon_wright #BCS_SIGIST Cloud Test Environments feat. VM Dispenser Technology Born in the Cloud service-oriented architecture
  16. 16. Summary - Test Automation as a Service (TaaaS.net) Relevant Clear traceability of the business value of Automation through the visualisation of the tests via Business Process Modelling (BPMNv2.2 compliant) Effective Self-validating test assets achieved using natural language with context sensitive validation against business and testing rules, workflows and data Maintainable Self-maintaining test asset loader/scraper Efficient Manageable Reports on SUT health including ratings such as percentage availability since build/release, reported errors over time and traffic to error ratio Unified platform which non-domain experts can use a natural language to represent business processes and user story acceptance criteria Portable Technology agonistic - Platform, client, component, browser, version & language Test type agnostic – smoke, regression, integration & performance Reliable Fault tolerance is built in to report and continue on different levels of fuzzy matching combined with the non-technology specific test definition language Actionable defects provided by environment under test (EUT) live pauseDiagnosable playback supported by dynamic data adapters (DDA) for accelerated defect investigation and resolution @jonathon_wright #BCS_SIGIST Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)
  17. 17. Panel Discussion - Should software testers be able to code? RIP Traditional Automation Frameworks (1993-2013) “Here lies beloved architects, engineers and developers who invested a disproportional amount of time and effort @jonathon_wright holy grail of test automation. Careers ended suddenly after a portfolio test manager searching for the www.sigist.org.uk #BCS_SIGIST asked them to provide the actual business value of what they were doing.”
  18. 18. Q1) How can we move to ‘Test Automation in the Cloud’? “Existing technologies of virtualisation, business process modelling, cloud based test automation solutions and rapid and easy @jonathon_wright internet access allow for the development of approaches that allow companies www.sigist.org.uk #BCS_SIGIST to order ‘Testing as a Service’ and pay only for what they use (PAYU).”
  19. 19. Automation Ready Cloud Power Test Automation in the Cloud @jonathon_wright #BCS_SIGIST Where are we going next?
  20. 20. Q2) Are we ready for ‘Test Automation in the Cloud’? Automation Maturity Model index (AMMi®) Assessment to provide a current automation maturity level and the target maturity level: Example Automation Projects AMMi Description Level 0 Phase Accelerating Automation Level Target Criteria Investment in Effort Business Benefit AMMi Gen Implementation Planning Execution Accel. ROE 4 5 24 63 17 6.9 112% Automated Test Lifecycle Methodology (ATLM) Client Specific Implementation - Hybrid Approach Framework Driven (Gen 1-5) Client Specific Implementation - BDD Approach 3 4 30 83 5.2 Processes are planned, performed, measured,22 and controlled Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26% Automation process defined and validated against international standards (ISO-29119) Level 1 Level 2 Level 3 Traditional Managed Sustaining Testing As A Service Model – Manual’mation Level 4 Quantified Testing As A Service Model - Hybrid Approach Level 5 Optimising TaaaS - Cloud Based Implementation 3 1 28 5 25 85% 4.6 200% 6.1 186% Predictability of automation process performance 4 5 12 31 19 Automation process variation and statistical predictability 5 5 5.6 25 7 16.0 306% * Source: AMMi®(http://www.AMMi.org.uk) + Test Automation Patterns (http://TestAutomationPatterns.wikispaces.com) “Sometimes half the battle just starting companies on their automation journey by taking them one step @jonathon_wright www.sigist.org.uk closer to becoming ready for automation.” AMMi Foundation 2003 #BCS_SIGIST
  21. 21. Q3) – Which ‘Automation Tool Platform’ “My current role at one of the worlds largest investment banks. Their landscape consisted of over 300+ solutions under test (SUT), representing 2,700+ applications under test (AUT), @jonathon_wright across 50+www.sigist.org.uk with over 300 automation practitioners worldwide.” global locations #BCS_SIGIST
  22. 22. Q3) – Which @jonathon_wright #BCS_SIGIST ‘Automation Platform’ Axe 4 (Beta2)
  23. 23. Quick Overview - ‘First Day Automation’ Approach Request.VM A1 C1 Login.Process A2 D1 D2 C2 C4 C3 Access.VM Logout.Process D3 Business Process Modelling (BPM) Business Process Scenarios (BPS) D4 E1 E2 A3 Business Process Test (BPT) E3 Manage.VM Business Process Data (BPD) Moment in time @jonathon_wright #BCS_SIGIST www.sigist.org.uk E4
  24. 24. Q4) How do we become ‘First Day Automation’ Ready? Start Static Source Cloud Tester Business Process Modelling Business Process Scenarios Agile Portfolio Management – BPM Export (Visio) Solution Lifecycle Management – Business Process Scenarios BPS Tool Modelling Tool (BizAgi or Visio) Business Process Tests @jonathon_wright #BCS_SIGIST (BizAgi or ALM/QC) BPT Tool @jonathon_wright #BCS_SIGIST www.eurostarconferences.com (UFT or Specification) HP Quality Centre (10+) / Application Lifecycle Management (11.5) Dynamic Source Solution Lifecycle Management - Business Process Scenarios Test Scheduler Direct Source Direct Source (xPDL or XAML) Solution Lifecycle Management – Business Process Testing @jonathon_wright #BCS_SIGIST (Workflow or XML) (WCF or REST) @jonathon_wright #BCS_SIGIST Cloud Source Agile Portfolio Management – Business Process Modelling Test Asset Loader (Test Asset Cube) Hyper-Test.com @jonathon_wright #BCS_SIGIST (Test Asset Journals) TaaaS.net (Test Designer)
  25. 25. APM Products Private Cloud Local Network SLM Products ALM Products Global-wide Portfolio Reporting Organisational Agile Enterprise Reporting Enterprise Agile Distributed Agile (Actionable Insight) Project Artefacts Blueprint Hybrid Cloud Executive Key Operating Procedures APM Products Dynamic Agile Processes Public Cloud Charter Q5) What is ‘Agile Portfolio Management’? Tier 0 – Programme Level (x4) Head of Automation Heads of Testing Global Technology Office Tier 1 - Workstream – Portfolio Level (x8) Portfolio Test Managers Workstream Leads Workstream Stakeholders Tier 2 - Domains – Solution Level (x330) Domain Test Managers Domain Leads Domain PM Domain Stakeholders Tier 3 - Sub-Domains – Application Level (x2,738) Vendors Development SD-PM SD-TEST SD-AUTOMATION “Everyday business and consumer life creates 2.5 quintillion bytes of data per day and 90% of the data in the world today has been created in the last two years?” with 1.5 petabytes of data across the current landscape.
  26. 26. Q6) How to provide ‘Actionable Insight / Automation Value’ Testing as a Service (TaaS) Dashboard (Realtime) Benefit Analysis Risk Dashboard Story Board (Presentation Layer) (Presentation Layer) (Presentation Layer) Business Stories Business Risk Business Domain Language Business Rules Project Risk Test Domain Language Business Workflow Technical Risk Business Questions Business Process Design (BPD) Non-Functional Requirements (NFR) Detailed Session Definition (DSD) (Visio Design Documentation) (Dynamic Data Source (XML)) (Dynamic Data Source (XAML)) Business Process Modelling (BPM) Business Components & Interfaces Detailed Profile Definition (DPD) (BPMNv2) (Dynamic Data Source (XAML)) (Dynamic Data Source (XAML)) Business Process Scenarios (BPS) Business Process Tests (BPT) Detailed Test Definition (DTD) (eXensible Process Definition Language) (Dynamic Data Source (XAML)) (Dynamic Data Source (XAML)) Business/Test Transactions Layer Dynamic Data Adapters @jonathon_wright #BCS_SIGIST Dynamic Test Adapters Solution Under Test (SUT) Layer www.sigist.org.uk
  27. 27. Dashboards (Real-time) providing ‘Actionable Insight’ @jonathon_wright #BCS_SIGIST Theme: Travel
  28. 28. Automation Ready Cloud Power Test Automation in the Cloud @jonathon_wright #BCS_SIGIST “To the cloud”
  29. 29. Q7) Should we move to the cloud? A match made in heaven? and/or New Challenges & Opportunities ? @jonathon_wright #BCS_SIGIST www.sigist.org.uk
  30. 30. New Challenges “For decades, RBS failed to invest properly in its systems” Ross McEwan, Chief Executive, RBS, December 3rd 2013 Alternative Service Provider? Unique Services (Such as Banks) "There's normally a role that technology plays in the outage, but when I look at the root cause, by far the greatest cause is people and processes." Damian Saunders, a cloud networking director What happens when these services FAIL? Connection Systemic Failure (Network or DNS) @jonathon_wright “Don’t assume the ecosystems (commercial or community) and #BCS_SIGIST fulfilmentDiagram Sourceto be tested Bankanything but isolation. -(http://www.newyorkfed.org/research/staff_reports/sr458.pdf) other systems - The Federal Reserve in of New York, Shadow Banking System, 2012 Julie Gardiner (2012) ”
  31. 31. New Opportunities Instant scalability Flexible Test Assets Flexible Test Execution Instant Accessibility Environments on-demand Geo-based execution 99.9% Consumer Freedom Availability (100% Web) CompareTesting.com Instant Portability No upfront investment “Enable sophisticated modelling of the components of cloud deployments, including servers, storage, databases and data. @jonathon_wright #BCS_SIGIST the consumption Pay Best of Breed (BoB) tools on demand of cloud resources” www.sigist.org.ukAs You Use (PAYU) For - PlanForCloud.com (2012)
  32. 32. Q8) Cloud Type Community Clouds SUT Private SUT SUT Combination Public TaaaS.net TaaaS.net Private Clouds Public Clouds “IBM are working with other the average enterprise will “... the prediction that 60% ofcloud suppliers to establish the have 60% of its applications in the cloud.” @jonathon_wright standards for cloud service provider compliance.” #BCS_SIGIST www.sigist.org.uk - Karl IBM Cloud Representative @Rhodes - Murat Gunenc,Flinders, Computer Weekly (2011)W1 (2011)
  33. 33. Automation Ready Cloud Power Test Automation in the Cloud @jonathon_wright #BCS_SIGIST The Final Frontier
  34. 34. Data Masking Community Cloud Data Protection Private Cloud Hybrid Cloud Infrastructure High Volume Automated Testing (HVAT) (as a Service) Applications Runtimes Lab Management Test Controller(s) High Volume Mobile Automated Testing (HVMAT) High Volume Mobile Automated Testing (HVMAT) Test Agent(s) Test Agent(s) – Telerik Art of Testing Framework Databases Servers Virtualization Storage Networking @jonathon_wright #BCS_SIGIST Public Cloud @jonathon_wright #BCS_SIGIST www.eurostarconferences.com Continuous Integration, Build & Deployment Data Controller – (Compliance with DPA) VMM Library Data Processor – (Written legal contract) VM Factory Data Provider – (European Economic Area) Q9) How can we move to ‘Test Automation in the Cloud’?
  35. 35. Q10) How can we execute ‘Test Automation in the Cloud’? Agile Portfolio Management – Business Process Data Agile Portfolio Management – Reporting - Test Execution Agile Portfolio Management – Reporting - Burndown Monitor velocity & capacity TFS Workbench @jonathon_wright #BCS_SIGIST ALTOVA XML Spy 2013 (R2 SP1) Microsoft Test Manager 2012.R3 @jonathon_wright #BCS_SIGIST www.eurostarconferences.com @jonathon_wright #BCS_SIGIST Business Process Reporting (BPR) Password Text Press Login Text Test Specific Language (TSL) Press | Login | Button “verbs and nouns in the test specific language are written in natural language which using context sensitive validation.” Verbs Nouns “The test editor is designed to be low waste, it will only offer nouns and verbs based on @jonathon_wright discovered artefacts not programming objects.” #BCS_SIGIST @jonathon_wright #BCS_SIGIST Business Process Tests (BPT) Agile Portfolio Management – Product Backlog Agile Portfolio Management – Agile Feature(s) Team Foundation Service (Product Backlog) Microsoft Project 2013 (Connected to TFS) Estimate Effort? @jonathon_wright #BCS_SIGIST Feature(s) Backlog Solution Lifecycle Management – Code Application Lifecycle Management – Unit Test(s) Team Foundation Service @jonathon_wright #BCS_SIGIST Business Process Scenarios (BPS) Agile Portfolio Management – Work Items > Tasks Agile Portfolio Management – Feature-Driven Agile Portfolio Management – Exploratory Testing Business Process Model (BPM) Agile Portfolio Management – Test Case(s) Analogue Digital @jonathon_wright #BCS_SIGIST Agile Portfolio Management – Business Process Modelling Button Events Username Enter Actions Object Enter Agile Portfolio Management – Business Process Scenarios Domain Specific Language Event Action Objects Login.Process Solution Lifecycle Management – Business Process Tests Business Process Data (BPD) TFS Workbench @jonathon_wright #BCS_SIGIST Work ID @jonathon_wright #BCS_SIGIST Feature-Driven Application Lifecycle Management – IntelliTrace Solution Lifecycle Management – Issue(s) @jonathon_wright #BCS_SIGIST www.eurostarconferences.com Exploratory Application Lifecycle Management – Action(s) Log @jonathon_wright #esconfs www.eurostarconferences.com Test Case(s) Application Lifecycle Management – Screen Capture(s) Test Case Migrator Plus (TFS) @jonathon_wright #BCS_SIGIST @jonathon_wright #BCS_SIGIST Unit Tests Code TFS - Gated Check-in OK Issue(s) www.eurostarconferences.com @jonathon_wright #BCS_SIGIST IntelliTrace VS2013 x64 Cross Tools Command Team Foundation Service ! @jonathon_wright #BCS_SIGIST Cancel Solution Lifecycle Management – Build Server Microsoft Windows [Version 8.1.9431] C:Automation>mstest /testlist:MasterDriver /testmetadata:”TestProject.vsmdi” /testsettings:”MobileDevices.run settings” @jonathon_wright #esconfs Action(s) Log Solution Under Test – Traffic Proxy ScreenCapture Solution Lifecycle Management - Interfaces Traffic Proxy Load Agent(s) – Visual Studio 2013 – Load Test Manager Interface(s) Load Agent(s) Solution Lifecycle Management – Test Agent(s) @jonathon_wright #BCS_SIGIST Test Studio for iOS (Telerik) Client(s) Cross Browser Testing – Telerik (Art of Testing - Framework) Database Server(s) @jonathon_wright #BCS_SIGIST Continuous Deployment @jonathon_wright #BCS_SIGIST www.eurostarconferences.com Test Controller(s) Shim/Stub(s) Application Under Test @jonathon_wright #BCS_SIGIST Solution Lifecycle Management – Test Controller(s) Application Server(s) Application Lifecycle Management – Client(s) Real Mobile Devices (PerfectoMobile) Build Server ASP .net | DynoForms (Ajax/Json/Spoc) @jonathon_wright #BCS_SIGIST @jonathon_wright #BCS_SIGIST @jonathon_wright #BCS_SIGIST Solution Lifecycle Management – Continuous Deployment @jonathon_wright #BCS_SIGIST @jonathon_wright #BCS_SIGIST Test Agent(s) www.eurostarconferences.com @jonathon_wright #BCS_SIGIST Browser(s) Solution Under Test Shim/Stub(s)
  36. 36. Final Thoughts - Test Automation in the Cloud “The best aspects of these proven test approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution – just as the underpinning technology evolves over time.” My section in the ‘Experiences in Test Automation’ book (Dorothy Graham & Mark Fewster) 2012. @jonathon_wright #BCS_SIGIST
  37. 37. Contact Details – Jonathon-Wright.com Testing-Alliance.com Testing Blog (www.Right-Testing.com) Automation Blog (www.TaaaS.net) @jonathon_wright #BCS_SIGIST www.sigist.org.uk LinkedIn + Email (jlwright@iee.org) www.linkedin.com/in/automation

×