Ha(API)y testing in the hybrid-cloud & beyond

513 views

Published on

Hybrid Automation (API) feat. YourTestCloud.com Testing in the Hybrid-Cloud & Beyond

A much overlooked and critical element of the modern day cloud ecosystem, is the interlocking web of dependencies often hidden and unknown to its varied clients and customers. For example, Google location services, or the Amazon eCommerce and fulfillment infrastructures are present in a myriad of services often at a level far removed from the core functionality.

The modern cloud aware, Service Oriented Architectures (SOA) are uniquely vulnerable to interlocking failure modes, caused by simple outages of services not perceived to be core or even present. The concept of private cloud computing has added an extra level of complexity to the Solution Under Test (SUT) ecosystem without solving any of the existing infrastructure challenges. Even normal operations of an interlocking web services can be perceived by an end user or client as an error.

The full value of both Messaging & API testing of complex ecosystems can be unlocked by utilising the power of Test Automation as a Service (TAaaS) platforms. These can be utilised to support testing of any ecosystem that has multiple hybrid cloud endpoints which span numerous Platform(s) / Infastructure(s) as a Service (PaaS / IaaS) providers.

Published in: Technology
  • Be the first to comment

Ha(API)y testing in the hybrid-cloud & beyond

  1. 1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Ha(API)y testing in the hybrid-cloud & beyond Hybrid Automation (API) Testing Jonathon Wright / “May the forth be with you”, 2014
  2. 2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2 non-Ha(API)y testing Currently we have over 5,000 end users of HP ALM across the company In a single ALM project we have 19,646 tests scheduled for manual execution Part of the current initiative we have targeted 2,700 applications across the company for automation The current process is that we walk through each test first before it can be automated
  3. 3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3 Each wave delivers automation for 250 projects across 100s of applications Once delivered automation executes against each build / release and can take over a week non-Ha(API)y testing The current initiative touches over 10,000 resources & stakeholders All have reporting requirements ranging from executive board level to individual teams practicing agile and traditional test methodologies
  4. 4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Adopting Global Agile Portfolio Management Board of Directors Exec Board VP (Portfolio) VP-1 (Domains) VP-2 (Apps) VP-3 (Projects) VP-4 (Teams) CIO ITPO ITDO ITAO ITPM ITTM CIO-1 ITPO-1 ITDO-1 ITAO-1 ITPM-1 ITTM-1 CIO-2 ITPO-2 ITDO-2 ITAO-2 ITPM-2 ITTM-2 CIO-3 ITPO-3 ITDO-3 ITAO-3 ITPM-3 ITTM-3 CIO-4 ITPO-4 ITDO-4 ITAO-4 ITPM-4 ITTM-4 Tier 0 - Programme – Executive Level (x4) Tier 1 - Workstreams – Portfolio Level (x8) Tier 2 - Domains – Solution Level (x330) Tier 3 - Sub-Domains – Application Level (x2,738) Tier 4 - Project-ID – Project Level (x7,324) Tier 5 - Staff-ID – Team Level (x10,000+)
  5. 5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 Adopting Ha(API)y testing 83% 6% 11% Automation (API) Unsuitable Unsupported 41% 33% 6% 20% Automation (UI) No UI Unsuitable Unsupported Native Support x2 Testing B(UI)ilds Ha(API)y Testing
  6. 6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 TESTING AS A SERVICE (TaaS) INFRASTRUCTURE AS A SERVICE (IaaS) SOFTWARE AS A SERVICE (SaaS) REAL-TIME VISIBILITY AND CONTROL INSTANT SCALABILITY SCALE UP AND DOWN HIGH AVAILABILITY CMMI PLATFORM AS A SERVICE (PaaS) ISO 29119PAY AS YOU USE (PAYU) SLA’s Adopting Cloud - Challenges & Opportunities “We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice?”
  7. 7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Adopting Cloud – Testing Challenges (UI) Focus shift right from physical (UI) Identity Services Connectivity (HTTPS) Workflows (Logic) DOM (HTML5) BIG Data (Database) External Services Workflows (Logic) • OCR / Image • Object Models • XPATH • Regular Expressions • Descriptive Programming • Fuzzy Logic • Artificial Intelligence • Social Intelligence
  8. 8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Adopting Cloud – Testing Opportunities (API) Focus shift right from physical (UI) to logical (API) / messaging layer / database layer API (SOA) AJAX CORBA AMF HTTP SOAP REST Database(s) ODBC Ole DB ADO .NET JDBC Java Native Messaging ESB AMQP MSMQ MQ JMS Tibco
  9. 9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Cloud Discovery – Learn physical endpoint(s) SOA Traditional functional testing focused on the physical (UI) endpoint Traditional non-functional testing focused on the messaging layer endpoint(s) Messaging Layer API SOA DBAPI Physical Server Endpoint Physical Client Endpoint Functional Non-Functional UI Ha(API)y
  10. 10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Cloud Discovery – Model logical endpoint(s) Endpoint discovery tool to map Cloud(s) “Globally, mobile phone users generate 5 petabytes of mobile data” petabytes [1pb equals 1,073,741,824mb] Source – Harvard School of MT Lab (2014) • Should we be adapting Network Fabric Models to map our Private Cloud(s)? • Imagine been asked as your first task to index 14,000 database endpoints?
  11. 11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Hybrid Cloud demands Hybrid Testing Manu’mation “Manu’mation is an approach to automated testing that leverages the manual effort involved in traditional testing to encapsulate both user and system under test interactions” Source – Test Automation Patterns - Wikispace (2014) Source - http://en.wikipedia.org/wiki/Hybrid_testing Source - Experiences of Test Automation – Book (2012) ‘First Day’ Testing “The key to ‘first day’ test automation is to create a dynamic test repository cube based on a combination of both fuzzy logic and descriptive languages (DSLs) to support the design and development of tests before the actual delivery of the Solution Under Test (SUT)”
  12. 12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 Manu'mation Adopting the ‘Test First Delivery’ (TFD) approach Manu’mation • Execute the manual tests via HP Sprinter • Capture test assets including associated injected data set • ‘Export to UFT’ as test artefacts that can be extracted in meta data (XML) Learn physical endpoint(s) • Define physical endpoint(s) • Learn (record) physical endpoint(s) • Simulate (playback) virtual endpoint(s) • Define (checkpoints) verification points • Validate (replay) virtual endpoint(s) Ha(API)y Testing • Leverage HP UFT API Testing • Enable Service Virtualisation • Execute tests in minutes not hours • Accurately report test results back into HP ALM against each build / release “Manu’mation is an approach to automated testing that leverages the manual effort involved in traditional testing to encapsulate both user and system under test interactions” Source – Test Automation Patterns - Wikispace (2014) http://testautomationpatterns.wikispaces.com/MANU%27MATION
  13. 13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 Manu'mation powered by HP Sprinter ‘Export to UFT’ plus associated data injection set
  14. 14. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. http://msdn.microsoft.com/en-us/bb906065.aspx Cloud Testing – Testing Challenges Focus shift from the private cloud network accessible with direct access via the messaging layer “A Message Bus is a combination of a common data model, a common command set, and a messaging infrastructure to allow different systems to communicate through a shared set of interfaces” “An enterprise service bus (ESB) is a software architecture model used for designing and implementing the interaction and communication between mutually interacting software applications in Service Oriented Architecture (SOA)” To secure (https) public cloud with limited access to the enterprise (internet) service bus
  15. 15. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 Cloud Testing – Learn physical endpoint(s) HP Service Virtualisation • Learns from the real endpoint(s) • Simulate response using recorded or user-defined response • Emulate endpoint(s) performance
  16. 16. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Application Server Rules Engine MQ API Data Warehouse Cloud Testing – Define virtual endpoint(s) Application Server Rules Engine MQ Discover Data Warehouse XML XML APIMQXML Test Execution (API) Service Virtualization Learn physical endpoint(s) Define virtual endpoint(s) Stub component(s) Shim method(s) Stub database(s) API
  17. 17. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17 Embracing Hybrid Cloud – Testing Challenges Introducing emerging complex cloud ecosystems (commercial or community) • Third-party public / private cloud endpoint(s) – Access to live public endpoints for testing? (test.paypal.com) – Access to private endpoints for testing? (VPN tunneling) • Security challenges around cloud platforms • Big Data (masking / protection / regulations) • Cloud 2.0 (moving away from legacy cloud) – BlackOps “data-in, data-out’ (DIDO) model
  18. 18. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Public VPN Tunnelling Private SUT SUTSUT Community Clouds Private Clouds Public Clouds VPN Endpoint VPN Endpoint Hybrid Cloud
  19. 19. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Hybrid Cloud Public Clouds Private Clouds Hybrid Cloud VPN Tunnelling Public VPN Endpoint Private VPN Endpoint SUT SUTSUT
  20. 20. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 Sync Systemic Failure (DNS or Network) Hybrid Cloud – Testing Challenges How do you provide realistic DR / resilience testing in the public cloud? “Don’t assume the ecosystems (commercial or community) and other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)
  21. 21. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21 ‘First Day’ Testing Adopting the ‘Test First Delivery’ (TFD) approach TFD >= ATDD/TDD/BDD • Create test definition(s) • Model the sequence of test event(s) • Build the Control Flow within UFT • Define (checkpoints) verification points Define virtual endpoint(s) • Define virtual endpoint(s) in SV • Define (mock) the rule(s) in SV • Generate (emulate) data-driven in SV • Simulate (playback) virtual endpoint(s) • Validate (replay) virtual endpoint(s) Ha(API)y Testing • Execute UFT BPT components • Enable Service Virtualisation • Execute tests without dependency of the Solution Under Test (SUT) • Leverage SV with HP LR “The key to ‘first day’ test automation is to create a dynamic test repository cube based on a combination of both fuzzy logic and descriptive languages to support the design and development of tests before the actual delivery of the Solution Under Test (SUT)” Source – Experiences of Test Automation (2012) http://www.dorothygraham.co.uk/automationExperiences
  22. 22. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22 ‘First Day’ - API Testing – Test Designer Drag-and-drop test elements to the canvas DB DBSOA DB DBEnterprise DB DBMiddleware DB DBSystem DB DBFlow Control DB DBCustom
  23. 23. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 ‘First Day’ powered by HP UFT (API Testing) vs. PCAP packet capture (promiscuous mode)Define virtual endpoint(s) Learn physical endpoint(s)
  24. 24. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24 Endpoint(s) Service Modelling ‘First Day’ powered by HP Service Virtualization Endpoint(s) Performance Modelling Endpoint(s) Data Modelling
  25. 25. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25 Leveraging the ‘First Day’ approach of ‘Test First Delivery’ (TFD) utilising HP Service Virtualisation & UFT (API) Enabled us to execute all of the tests without the dependency of the Solution Under Test (SUT) from the very ‘first day’ in Sprint 0. Ha(API)y testing Adopting the ‘Manu’mation’ approach of ‘Test First Delivery’ (TFD) utilising HP Sprinter Enabled us to capture 19,646 tests over 4 major releases across 12 sprints.
  26. 26. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. select <IaaS> select <PaaS> select <SaaS> select <TaaS> select <TaaaS> Cloud Recipe for Solution Under Test Cloud Recipe (IFTTT) Cloud 2.0 – Moving beyond the legacy cloud “IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.” - Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011) CloudSystemAutomation YourTestCloud.com “Heterogeneous test cloud that are open, flexible and extensible” Testing Cloud If <Solution Under Test> Then <Build Cloud 2.0> Testing Cloud
  27. 27. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Final Thoughts Thank you “The best aspects of these proven approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution – just as the underpinning technology evolves over time.” My section in the ‘Experiences in Test Automation’ book (Dorothy Graham & Mark Fewster) 2012.

×