Gartner - Ha(API)y testing in the hybrid-cloud & beyond

2,178 views

Published on

Hybrid Automation (API) feat. YourTestCloud.com Testing in the Hybrid-Cloud & Beyond

A much overlooked and critical element of the modern day cloud ecosystem, is the interlocking web of dependencies often hidden and unknown to its varied clients and customers. For example, Google location services, or the Amazon eCommerce and fulfillment infrastructures are present in a myriad of services often at a level far removed from the core functionality.

The modern cloud aware, Service Oriented Architectures (SOA) are uniquely vulnerable to interlocking failure modes, caused by simple outages of services not perceived to be core or even present. The concept of private cloud computing has added an extra level of complexity to the Solution Under Test (SUT) ecosystem without solving any of the existing infrastructure challenges. Even normal operations of an interlocking web services can be perceived by an end user or client as an error.

The full value of both Messaging & API testing of complex ecosystems can be unlocked by utilising the power of Test Automation as a Service (TAaaS) platforms. These can be utilised to support testing of any ecosystem that has multiple hybrid cloud endpoints which span numerous Platform(s) / Infastructure(s) as a Service (PaaS / IaaS) providers.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,178
On SlideShare
0
From Embeds
0
Number of Embeds
997
Actions
Shares
0
Downloads
30
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Gartner - Ha(API)y testing in the hybrid-cloud & beyond

  1. 1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Ha(API)y testing in the hybrid-cloud & beyond Hybrid Automation (API) Testing Jonathon Wright / 19th May, 2014 Emerging Technologies, Innovation & Automation Gartner ADDI (SPS2):
  2. 2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2 non-Ha(API)y testing Currently we have over 5,000 end users of HP ALM across the company In a single ALM project we have 19,646 tests scheduled for manual execution Part of the current initiative we have targeted 2,700 applications across the company for automation The current process is that we walk through each test first before it can be automated
  3. 3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3 Each wave delivers automation for 250 projects across 100s of applications Once delivered automation executes against each build / release and can take over a week non-Ha(API)y testing The current initiative touches over 10,000 resources & stakeholders All have reporting requirements ranging from executive board level to individual teams practicing agile and traditional test methodologies
  4. 4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Adopting Global Agile Portfolio Management Board of Directors Exec Board VP (Portfolio) VP-1 (Domains) VP-2 (Apps) VP-3 (Projects) VP-4 (Teams) CIO ITPO ITDO ITAO ITPM ITTM CIO-1 ITPO-1 ITDO-1 ITAO-1 ITPM-1 ITTM-1 CIO-2 ITPO-2 ITDO-2 ITAO-2 ITPM-2 ITTM-2 CIO-3 ITPO-3 ITDO-3 ITAO-3 ITPM-3 ITTM-3 CIO-4 ITPO-4 ITDO-4 ITAO-4 ITPM-4 ITTM-4 Tier 0 - Programme – Executive Level (x4) Tier 1 - Workstreams – Portfolio Level (x8) Tier 2 - Domains – Solution Level (x330) Tier 3 - Sub-Domains – Application Level (x2,738) Tier 4 - Project-ID – Project Level (x7,324) Tier 5 - Staff-ID – Team Level (x10,000+)
  5. 5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 Adopting Ha(API)y testing 83% 6% 11% Automation (API) Unsuitable Unsupported 41% 33% 6% 20% Automation (UI) No UI Unsuitable Unsupported Native Support x2 Testing B(UI)ilds Ha(API)y Testing Sample – 2,738 applications under test (AUTs)
  6. 6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 TESTING AS A SERVICE (TaaS) INFRASTRUCTURE AS A SERVICE (IaaS) SOFTWARE AS A SERVICE (SaaS) REAL-TIME VISIBILITY AND CONTROL INSTANT SCALABILITY SCALE UP AND DOWN HIGH AVAILABILITY CMMI PLATFORM AS A SERVICE (PaaS) ISO 29119PAY AS YOU USE (PAYU) SLA’s Adopting Cloud - Challenges & Opportunities “We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice?”
  7. 7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Adopting Cloud – FrAgile (UI) Testing Focus shift right from physical (UI) Identity Services Connectivity (HTTPS) Workflows (Logic) DOM (HTML5) BIG Data (Database) External Services Workflows (Logic) • OCR / Image • Object Models • XPATH • Regular Expressions • Descriptive Programming • Fuzzy Logic • Social Intelligence • Artificial Intelligence www.slideshare.net/jonathon_wright
  8. 8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Adopting Cloud – Testing Opportunities (API) Focus shift right from physical (UI) to logical (API) / messaging layer / database layer SOA (API) AJAX CORBA AMF HTTP SOAP REST Database(s) ODBC Ole DB ADO .NET JDBC Java Native Messaging ESB AMQP MSMQ MQ JMS Tibco
  9. 9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Cloud Discovery – Learn physical endpoint(s) SOA Traditional functional testing focused on the physical (UI) endpoint Traditional non-functional testing focused on the messaging layer endpoint(s) Messaging Layer API SOA DBAPI Physical Server Endpoint Physical Client Endpoint Functional Non-Functional UI Ha(API)y Testing
  10. 10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Cloud Discovery – Model logical endpoint(s) Endpoint discovery Cloud Map modelling “Globally, mobile phone users generate 5 petabytes of mobile data” petabytes [1pb equals 1,073,741,824mb] Source – Harvard School of MT Lab (2014) • Should we be adapting Network Fabric Models to map our Private Cloud(s)? • Imagine been asked as your first task to index 14,000 database endpoints?
  11. 11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Embracing Hybrid Cloud – Testing Challenges Introducing emerging complex cloud ecosystems (commercial or community) • Third-party public / private cloud endpoint(s) – Access to live public endpoints for testing? (test.paypal.com) – Access to private endpoints for testing? (VPN tunneling) • Security challenges around cloud platforms • Big Data (masking / protection / regulations) • Cloud 2.0 (moving away from legacy cloud) – BlackOps “data-in, data-out’ (DIDO) model
  12. 12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 Hybrid Cloud demands Hybrid Testing Manu’mation “Manu’mation is an approach to automated testing that leverages the manual effort involved in traditional testing to encapsulate both user and system under test interactions” Source – Test Automation Patterns - Wikispace (2014) Source - http://en.wikipedia.org/wiki/Hybrid_testing Source - Experiences of Test Automation – Book (2012) ‘First Day’ Testing “The key to ‘first day’ test automation is to create a dynamic test repository cube based on a combination of both fuzzy logic and descriptive languages (DSLs) to support the design and development of tests before the actual delivery of the Solution Under Test (SUT)”
  13. 13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 Manu'mation Adopting the ‘Test First Delivery’ (TFD) approach Manu’mation • Execute the manual tests via HP Sprinter • Capture test assets including associated injected data set • ‘Export to UFT’ as test artefacts that can be extracted in meta data (XML) Learn physical endpoint(s) • Define physical endpoint(s) • Learn (record) physical endpoint(s) • Simulate (playback) virtual endpoint(s) • Define (checkpoints) verification points • Validate (replay) virtual endpoint(s) Ha(API)y Testing • Leverage HP UFT API Testing • Enable Service Virtualisation • Execute tests in minutes not hours • Accurately report test results back into HP ALM against each build / release “Manu’mation is an approach to automated testing that leverages the manual effort involved in traditional testing to encapsulate both user and system under test interactions” Source – Test Automation Patterns - Wikispace (2014) http://testautomationpatterns.wikispaces.com/MANU%27MATION
  14. 14. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 Manu'mation powered by HP Sprinter ‘Export to UFT’ plus associated data injection set
  15. 15. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. http://msdn.microsoft.com/en-us/bb906065.aspx Cloud Testing – Testing Challenges Focus shift from the private cloud network accessible with direct access via the messaging layer “An enterprise service bus (ESB) is a software architecture model used for designing and implementing the interaction and communication between mutually interacting software applications in Service Oriented Architecture (SOA)” To secure (https) public cloud with limited access to the enterprise (internet) service bus
  16. 16. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Cloud Testing – Learn physical endpoint(s) HP Service Virtualisation • Learns from the real endpoint(s) • Simulate response using recorded or user-defined response • Emulate endpoint(s) performance
  17. 17. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Public VNet-to-VNet Private SUT SVNV NVSV Community Clouds Private Clouds Public Clouds VNet Endpoint VNet Endpoint Hybrid Cloud
  18. 18. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Hybrid Cloud Public Clouds Private Clouds Hybrid Cloud VNet-to-VNet Public VNet Endpoint Private VNet Endpoint RRAS/VPN/NAT connections OSI 3 - Network - IPSec OSI 5 - Session - PPTP OSI 7 - Application - SSL VPN RRAS/VPN/NAT communication FTP - 21 HTTP - 80 RPC - 135 CIFS - 139/445 Source - http://msdn.microsoft.com/en-us/library/azure/jj156007.aspx SUT SVNV NVSV
  19. 19. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19 Sync Systemic Failure (DNS or Network) Hybrid Cloud – Testing Challenges How do you provide realistic DR / resilience testing in the public cloud? “Don’t assume the ecosystems (commercial or community) and other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)
  20. 20. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 Application Server Rules Engine MQ API Small Data Cloud Testing – Define virtual endpoint(s) Application Server Rules Engine MQ Discover Big Data XML XML APIMQXML Test Execution (API) Service Virtualization Learn physical endpoint(s) Define virtual endpoint(s) Stub component(s) Shim method(s) Stub database(s) API
  21. 21. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21 ‘First Day’ Testing Adopting the ‘Test First Delivery’ (TFD) approach TFD >= ATDD/TDD/BDD • Create test definition(s) • Model the sequence of test event(s) • Build the Control Flow within UFT • Define (checkpoints) verification points Define virtual endpoint(s) • Define virtual endpoint(s) in SV • Define (mock) the rule(s) in SV • Generate (emulate) data-driven in SV • Simulate (playback) virtual endpoint(s) • Validate (replay) virtual endpoint(s) Ha(API)y Testing • Execute UFT BPT components • Enable Service Virtualisation • Execute tests without dependency of the Solution Under Test (SUT) • Leverage SV with HP LR “The key to ‘first day’ test automation is to create a dynamic test repository cube based on a combination of both fuzzy logic and descriptive languages to support the design and development of tests before the actual delivery of the Solution Under Test (SUT)” Source – Experiences of Test Automation (2012) http://www.dorothygraham.co.uk/automationExperiences
  22. 22. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22 ‘First Day’ - API Testing – Test Designer Drag-and-drop test elements to the canvas DB DBSOA DB DBEnterprise DB DBMiddleware DB DBSystem DB DBFlow Control DB DBCustom
  23. 23. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 Endpoint(s) Service Modelling ‘First Day’ powered by HP Service Virtualization Endpoint(s) Performance Modelling Endpoint(s) Data Modelling
  24. 24. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24 ‘First Day’ powered by HP Network Virtualization Virtualize network conditions
  25. 25. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25 ‘First Day’ powered by HP UFT (API Testing) vs. PCAP packet capture (promiscuous mode)Define virtual endpoint(s) Learn remote physical endpoint(s)
  26. 26. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26 Leveraging the ‘First Day’ approach of ‘Test First Delivery’ (TFD) utilising HP Service Virtualisation & UFT (API) Enabled us to execute all of the tests without the dependency of the Solution Under Test (SUT) from day-zero. Ha(API)y testing - Summary Adopting the ‘Manu’mation’ approach of ‘Test First Delivery’ (TFD) utilising HP Sprinter Enabled us to capture 19,646 tests over 4 major releases across 12 sprints.
  27. 27. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Ha(API)y testing – Takeaways • UI Automation is Costly & FrAgile • API Automation is Repeatable, Reliable and Fast • Create Bridges (VNet to VNet) “If you build it, they will test” • Use Cloud Maps to Learn, Discover and Model endpoint(s) • Enable Day-zero Performance, Penetration & Security Testing • Continuous Build, Integration & Delivery in YourTestCloud • Embrace Community Test Clouds (Collaborate/Share Recipes) Testing Recipes (IFTTT) If <solution under test> exists Then <Learn Physical Endpoints> Else If <solution under test> does not exist Then <Define Virtual Endpoints>
  28. 28. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. select <IaaS> select <PaaS> select <SaaS> select <TaaS> select <TaaaS> Cloud Recipe for Solution Under Test Cloud Recipe (IFTTT) Cloud 2.0 – Moving beyond the legacy cloud “IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.” - Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011) CloudSystemAutomation YourTestCloud.com “Heterogeneous test cloud that are open, flexible and extensible” Testing Cloud If <Solution Under Test> Then <Build Cloud 2.0> Testing Cloud
  29. 29. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Final Thoughts Thank you “The best aspects of these proven approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution – just as the underpinning technology evolves over time.” My section in the ‘Experiences in Test Automation’ book (Dorothy Graham & Mark Fewster) 2012.

×