As the complexity of interaction between software serving different purposes is evolving with cloud computing and virtualization, so is the need for validation solutions and supporting methodologies. Conventional test tools were not designed to test “hypervisors”—the cornerstone of virtualization.

1. Cloud Testing with Synthetic
Workload Generators
White Paper
Why cloud testing?
Cloud Computing and Virtualization (NV and NFV) are adding incredible diversity in the
number of providers and services offered, but most of this ecosystem is running on a
handful of virtual platforms from vendors such as VMware, Citrix, Amazon, and Microsoft,
or from a purely open source solution such as KVM. Compared to traditional server
applications and networking products however, virtual platforms are comparatively new
and obviously less tested. This point can be derived from the lack of test tools targeted
to testing virtualization platforms. There is a burgeoning new class of cloud test tools
referred to as “Cloud Management Platforms or CMP tools”; but these tools focus more
on capacity planning, active monitoring, and basic network performance within a public,
private or hybrid cloud than they do actual benchmarking of the virtual platform at the
center of cloud or NFV infrastructure. CMP tools also derive the majority of their analytics
from native APIs provided by the hypervisor, duplicating information already offered by
hypervisor vendor tools. There are also cloud based traffic generators, SaaS offerings
that specifically aim to determine application traffic performance over the cloud, but
such tools are limited in their scope of test methodologies and even the analytics they
provide are only about network performance and not virtual infrastructure.
Introduction
Cloud computing is redefining the meaning
of scale and complexity. It also demands the
ability to fuse the previously distinct roles
of application hosting and network traffic
processing over a common system. Until
recently, application testing and network device
testing are treated as two very distinct domains
starting from different hardware and software
to different testing tools and practices. As the
complexity of interaction between software
serving different purposes is evolving with
cloud computing and virtualization, so is the
need for validation solutions and supporting
methodologies. Conventional test tools were
not designed to test “hypervisors” – the
cornerstone of virtualization. Spirent intends to
introduce the concept of a complete resource
bearing “synthetic workload generator” as
the best testing solution for benchmarking
hypervisors – by validating both functionality
and performance of various applications and
services (IaaS, PaaS, SaaS) running over them.

2. Cloud Testing with Synthetic Workload Generators
2 | spirent.com
White Paper
Organic vs. Synthetic Workloads
The lack of adequate test tools targeting virtualization platforms can largely be attributed to the complexity of developing such
tools. The current approach is to use “organic” workloads where actual applications create the workloads and traffic running on
a virtual machine, e.g. web server, then generate client side requests via automation to scale up the workloads and traffic. This
approach is effective, but can be very costly to scale for software that is not open source. The “organic” approach also requires
that software and its dependencies such as drivers, packages etc., be installed which introduces considerable overhead testing
from one type of application to another. This significantly adds to the time and complexity to execute testing. Lastly, organic
workloads are not flexible enough. Workloads are confined to the natural state machine of the application under test. Complex
defects only surface when the test loads run for a lengthy period of time, or if the test loads can break out of the confinement of a
particular application’s state machine. Organic workloads cannot have a particular resource like CPU, memory, or storage, skewed
to change the load characteristic of the test. A better approach pioneered by Spirent is the concept of “synthetic workloads”.
Synthetic workloads will drive the utilization of vCPUs, memory, storage I/O, and network I/O using synthetically created processes
and threads of execution at the VM level for hypervisor scheduling. Using synthetic workloads is a far less expensive approach
that scales higher, automates easier, offers much greater workload flexibility, and ultimately exposes more bottlenecks in resource
scheduling by the hypervisor kernel – ultimately improving the performance of virtual platforms that SaaS and virtual service chains
run on.
Introduction to Synthetic Workload Generators
Just as the popularity and scale of the Internet mandated the need for new types of test tools that could generate and receive
hundreds of millions of IP flows, and emulate hundreds of thousands of IP switching and routing devices, tools Spirent introduced
to the world and has been a leader in ever since. So too, will Cloud Computing necessitate a new generation of tools be developed
to meet the sheer scale and diversity that complex deployments will demand of the virtual platforms they are built on. Synthetic
Workload Generators (SWGs) not only can provide the cost and scale benefits, but also control, flexibility, load isolation, and real-
time feedback that unaltered organic application workloads fall short of. SWGs are designed for live production environments and
deployed in actual live cloud or virtual infrastructure. Key features include:
ƒƒ Ability to provide any type of modeled workload on a single resource, any combination of resources, or even all compute
resources simultaneously providing unparalleled control and flexibility of workload scheduling
ƒƒ Cause adversity or havoc like sustained high CPU utilization or excessive memory consumption emulating memory leak
conditions
ƒƒ Provide deep insights into a hypervisor’s scheduling algorithm performance and ability to respond to such problems
ƒƒ Help preempt service deployment issues, isolate faulty hardware in the cloud, and expose complex defects with the
hypervisor platform that IaaS, PaaS, SaaS, or NFV is running over
ƒƒ Ability to gauge or conduct dry runs of a particular service before deploying the actual cloud or NFV application and quickly
throttle back from workload thresholds that impact live production VMs.
These features add up to a complete divergence from conventional network testing where load generators are connected to lab
test beds or application performance is assessed on non-production servers.

3. spirent.com | 3
Use cases for Synthetic Workload Generators
By utilizing VMs or containers to generate and precisely control the magnitude of, per resource synthetic workloads – vCPU,
memory, storage I/O, network I/O – and derive real-time feedback on the performance of each independent workload operation,
any type of test can be run cost effectively, in any environment whether live or not, and can provide a complete performance
assessment over any period of time for a given virtual infrastructure. For example, individual resource control allows for workload
isolation to just a specific resource operation, memory read for example. This load isolation can then be scaled across many
generators, also referred to as agents, to determine the overall scale capability of a given resource operation like memory reads
in a cloud or NFV deployment. Organic software loads cannot mimic this behavior. Cloud and Service Providers could use this
information to better gauge their VM memory capacity planning. SWGs can be used to validate the QoE or SLAs of virtualized
services and their performance under scale across all resources. Cloud customers, or even private cloud owners, could use such
a tool to determine the best provider or hypervisor platform to build their cloud on based on a given IaaS, PaaS, SaaS, or NFV
implementation.
Figure:
Bare Metal Server
HypervisorDeployment
Service
Analytics
Collection
SWG1
SWG2
SWG3
SWG4
SWGn
Synthetic Workload Generators deployed for auto-scale use case
Synthetic Workload Generators can model custom enterprise applications that are impractical to obtain or implement for testing or
benchmarking purposes. SWGs can also be used to accurately assess spare capacity, which is especially critical to smaller private
cloud implementations that need to maximize hardware resources, or Cloud/Service Providers trying to assess the impact of
adding another service chain to compute nodes already hosting other tenants. SWGs are perfectly suited to auto scale validations,
whether assessing boot time, policy enforcement or determining the maximum achievable scale of an elastic computing solution in
a large public cloud, providing pinpoint analytics of every operation across all resources per agent exposing resource contention
issues and bottlenecks at scales that are difficult to comprehend. These examples are just the tip of the iceberg; many more use
cases exist in areas like high availability and migration performance benchmarking.

4. © 2016 Spirent. All Rights Reserved.
All of the company names and/or brand names and/or product names referred to in this document, in particular, the name “Spirent”
and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in
accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners.
The information contained in this document is subject to change without notice and does not represent a commitment on the part
of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or
liability for any errors or inaccuracies that may appear in the document. Rev A | 01/16
Cloud Testing with Synthetic Workload Generators
spirent.com
AMERICAS 1-800-SPIRENT
+1-800-774-7368 | sales@spirent.com
EUROPE AND THE MIDDLE EAST
+44 (0) 1293 767979 | emeainfo@spirent.com
ASIA AND THE PACIFIC
+86-10-8518-2539 | salesasia@spirent.com
White Paper
Summary
Cloud computing and all of its instantiations require a new type of tool to validate
the platforms they run over. A tool that is commercially available, economical, meets
a hyperscale requirement, and flexible enough to simulate any type of application
workload, one that allows precise control and provides key analytics plus offers real-
time feedback on compute resource performance and that is driven by methodologies
applicable to cloud and NFV applications. Synthetic Workload Generators meet all of
these requirements. Synthetic application workloads are not a new concept. It has also
been proven that application workloads can be accurately and realistically synthesized1
.
And Spirent is pioneering efforts towards bringing such tools and supporting
methodologies to market. For more information about Spirent Cloud Solutions, please
visit www.spirent.com/Cloud.
About Spirent
Spirent provides software solutions to high-tech
equipment manufacturers and service providers
that simplify and accelerate device and system
testing. Developers and testers create and share
automated tests that control and analyze results
from multiple devices, traffic generators, and
applications while automatically documenting
each test with pass-fail criteria. With Spirent
solutions, companies can move along the path
toward automation while accelerating QA cycles,
reducing time to market, and increasing the
quality of released products. Industries such
as communications, aerospace and defense,
consumer electronics, automotive, industrial,
and medical devices have benefited from
Spirent products.
1 Georgia Institute of Technology: Synthetic Workload Generation for Cloud Computing
Applications, Bahga and Krishna Madisetti