Cloud testing with synthetic workload generators

1. Cloud Testing with Synthetic Workload Generators White Paper Why cloud testing? Cloud Computing and Virtualization (NV and NFV) are adding incredible diversity in the number of providers and services offered, but most of this ecosystem is running on a handful of virtual platforms from vendors such as VMware, Citrix, Amazon, and Microsoft, or from a purely open source solution such as KVM. Compared to traditional server applications and networking products however, virtual platforms are comparatively new and obviously less tested. This point can be derived from the lack of test tools targeted to testing virtualization platforms. There is a burgeoning new class of cloud test tools referred to as “Cloud Management Platforms or CMP tools”; but these tools focus more on capacity planning, active monitoring, and basic network performance within a public, private or hybrid cloud than they do actual benchmarking of the virtual platform at the center of cloud or NFV infrastructure. CMP tools also derive the majority of their analytics from native APIs provided by the hypervisor, duplicating information already offered by hypervisor vendor tools. There are also cloud based traffic generators, SaaS offerings that specifically aim to determine application traffic performance over the cloud, but such tools are limited in their scope of test methodologies and even the analytics they provide are only about network performance and not virtual infrastructure. Introduction Cloud computing is redefining the meaning of scale and complexity. It also demands the ability to fuse the previously distinct roles of application hosting and network traffic processing over a common system. Until recently, application testing and network device testing are treated as two very distinct domains starting from different hardware and software to different testing tools and practices. As the complexity of interaction between software serving different purposes is evolving with cloud computing and virtualization, so is the need for validation solutions and supporting methodologies. Conventional test tools were not designed to test “hypervisors” – the cornerstone of virtualization. Spirent intends to introduce the concept of a complete resource bearing “synthetic workload generator” as the best testing solution for benchmarking hypervisors – by validating both functionality and performance of various applications and services (IaaS, PaaS, SaaS) running over them.

2. Cloud Testing with Synthetic Workload Generators 2 | spirent.com White Paper Organic vs. Synthetic Workloads The lack of adequate test tools targeting virtualization platforms can largely be attributed to the complexity of developing such tools. The current approach is to use “organic” workloads where actual applications create the workloads and traffic running on a virtual machine, e.g. web server, then generate client side requests via automation to scale up the workloads and traffic. This approach is effective, but can be very costly to scale for software that is not open source. The “organic” approach also requires that software and its dependencies such as drivers, packages etc., be installed which introduces considerable overhead testing from one type of application to another. This significantly adds to the time and complexity to execute testing. Lastly, organic workloads are not flexible enough. Workloads are confined to the natural state machine of the application under test. Complex defects only surface when the test loads run for a lengthy period of time, or if the test loads can break out of the confinement of a particular application’s state machine. Organic workloads cannot have a particular resource like CPU, memory, or storage, skewed to change the load characteristic of the test. A better approach pioneered by Spirent is the concept of “synthetic workloads”. Synthetic workloads will drive the utilization of vCPUs, memory, storage I/O, and network I/O using synthetically created processes and threads of execution at the VM level for hypervisor scheduling. Using synthetic workloads is a far less expensive approach that scales higher, automates easier, offers much greater workload flexibility, and ultimately exposes more bottlenecks in resource scheduling by the hypervisor kernel – ultimately improving the performance of virtual platforms that SaaS and virtual service chains run on. Introduction to Synthetic Workload Generators Just as the popularity and scale of the Internet mandated the need for new types of test tools that could generate and receive hundreds of millions of IP flows, and emulate hundreds of thousands of IP switching and routing devices, tools Spirent introduced to the world and has been a leader in ever since. So too, will Cloud Computing necessitate a new generation of tools be developed to meet the sheer scale and diversity that complex deployments will demand of the virtual platforms they are built on. Synthetic Workload Generators (SWGs) not only can provide the cost and scale benefits, but also control, flexibility, load isolation, and real- time feedback that unaltered organic application workloads fall short of. SWGs are designed for live production environments and deployed in actual live cloud or virtual infrastructure. Key features include: Ability to provide any type of modeled workload on a single resource, any combination of resources, or even all compute resources simultaneously providing unparalleled control and flexibility of workload scheduling Cause adversity or havoc like sustained high CPU utilization or excessive memory consumption emulating memory leak conditions Provide deep insights into a hypervisor’s scheduling algorithm performance and ability to respond to such problems Help preempt service deployment issues, isolate faulty hardware in the cloud, and expose complex defects with the hypervisor platform that IaaS, PaaS, SaaS, or NFV is running over Ability to gauge or conduct dry runs of a particular service before deploying the actual cloud or NFV application and quickly throttle back from workload thresholds that impact live production VMs. These features add up to a complete divergence from conventional network testing where load generators are connected to lab test beds or application performance is assessed on non-production servers.

3. spirent.com | 3 Use cases for Synthetic Workload Generators By utilizing VMs or containers to generate and precisely control the magnitude of, per resource synthetic workloads – vCPU, memory, storage I/O, network I/O – and derive real-time feedback on the performance of each independent workload operation, any type of test can be run cost effectively, in any environment whether live or not, and can provide a complete performance assessment over any period of time for a given virtual infrastructure. For example, individual resource control allows for workload isolation to just a specific resource operation, memory read for example. This load isolation can then be scaled across many generators, also referred to as agents, to determine the overall scale capability of a given resource operation like memory reads in a cloud or NFV deployment. Organic software loads cannot mimic this behavior. Cloud and Service Providers could use this information to better gauge their VM memory capacity planning. SWGs can be used to validate the QoE or SLAs of virtualized services and their performance under scale across all resources. Cloud customers, or even private cloud owners, could use such a tool to determine the best provider or hypervisor platform to build their cloud on based on a given IaaS, PaaS, SaaS, or NFV implementation. Figure: Bare Metal Server HypervisorDeployment Service Analytics Collection SWG1 SWG2 SWG3 SWG4 SWGn Synthetic Workload Generators deployed for auto-scale use case Synthetic Workload Generators can model custom enterprise applications that are impractical to obtain or implement for testing or benchmarking purposes. SWGs can also be used to accurately assess spare capacity, which is especially critical to smaller private cloud implementations that need to maximize hardware resources, or Cloud/Service Providers trying to assess the impact of adding another service chain to compute nodes already hosting other tenants. SWGs are perfectly suited to auto scale validations, whether assessing boot time, policy enforcement or determining the maximum achievable scale of an elastic computing solution in a large public cloud, providing pinpoint analytics of every operation across all resources per agent exposing resource contention issues and bottlenecks at scales that are difficult to comprehend. These examples are just the tip of the iceberg; many more use cases exist in areas like high availability and migration performance benchmarking.

4. © 2016 Spirent. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name “Spirent” and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document. Rev A | 01/16 Cloud Testing with Synthetic Workload Generators spirent.com AMERICAS 1-800-SPIRENT +1-800-774-7368 | sales@spirent.com EUROPE AND THE MIDDLE EAST +44 (0) 1293 767979 | emeainfo@spirent.com ASIA AND THE PACIFIC +86-10-8518-2539 | salesasia@spirent.com White Paper Summary Cloud computing and all of its instantiations require a new type of tool to validate the platforms they run over. A tool that is commercially available, economical, meets a hyperscale requirement, and flexible enough to simulate any type of application workload, one that allows precise control and provides key analytics plus offers real- time feedback on compute resource performance and that is driven by methodologies applicable to cloud and NFV applications. Synthetic Workload Generators meet all of these requirements. Synthetic application workloads are not a new concept. It has also been proven that application workloads can be accurately and realistically synthesized1 . And Spirent is pioneering efforts towards bringing such tools and supporting methodologies to market. For more information about Spirent Cloud Solutions, please visit www.spirent.com/Cloud. About Spirent Spirent provides software solutions to high-tech equipment manufacturers and service providers that simplify and accelerate device and system testing. Developers and testers create and share automated tests that control and analyze results from multiple devices, traffic generators, and applications while automatically documenting each test with pass-fail criteria. With Spirent solutions, companies can move along the path toward automation while accelerating QA cycles, reducing time to market, and increasing the quality of released products. Industries such as communications, aerospace and defense, consumer electronics, automotive, industrial, and medical devices have benefited from Spirent products. 1 Georgia Institute of Technology: Synthetic Workload Generation for Cloud Computing Applications, Bahga and Krishna Madisetti