Harnessing the Cloud for Performance Testing- Impetus White Paper


Published on

For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper

The paper provides insights on the various benefits of using the Cloud for Performance Testing as well as how to address the various challenges associated with this approach.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Harnessing the Cloud for Performance Testing- Impetus White Paper

  1. 1. Harnessing the Cloud forPerformance TestingW H I T E P A P E RAbstractIn this white paper, Impetus talks about how the infinitecomputing power of a Cloud infrastructure can be leveraged forflexible, scalable and affordable load testing.The paper provides insights on the various benefits of using theCloud for Performance Testing as well as how to address thevarious challenges associated with this approach. It discusses thedifferent models where performance testing tools can bedeployed on the Cloud, in addition to some best practices andreal-world experiences.Impetus Technologies, Inc.www.impetus.com
  2. 2. Harnessing the Cloud for Performance Testing2Table of ContentsIntroduction...........................................................................................................3The challenges and consequences of the growth in data and the user base.......4Harnessing the power of the Cloud.......................................................................5The flip side of Cloud and performance testing ....................................................6The tool deployment models for Cloud-based performance testing ...................7The Impetus solution...........................................................................................10Summary..............................................................................................................11
  3. 3. Harnessing the Cloud for Performance Testing3IntroductionCloud-based performance testing has suddenly become a major buzzwordtoday. From startups, to large enterprises, everyone is trying to make the mostof the opportunities offered by the Cloud.The fact is that that there are several distinct benefits that Cloud infrastructureoffers organizations. It enables faster-time-to-test, is highly scalable and an on-demand solution. The Cloud also reduces the overall performance testing costs.It is important however, to understand that why the Cloud is emerging as animportant paradigm in performance testing.Trends show that the Internet is growing at an unprecedented rate. Globallythere are close to two billion internet users, with Facebook alone crossing the500 million user mark and Twitter receiving millions of tweets every month. Theuse of the Web and mobile applications has also increased phenomenally, withthe availability of devices like smartphones, netbooks and iPads.This scenario, while beneficial for end users, is creating several challenges forapplication architects and developers, who have to ensure that applications areperforming efficiently and effectively under this huge, increasing load. Theconcern is not only the two billion users on the Internet, but also the increase inthe size of the data.Today, data sizes within organizations have gone up from hundreds of millionsof rows, to billions of rows per day.InternetGrowthBig Data Challenges andConsequencesMoving to Cloud
  4. 4. Harnessing the Cloud for Performance Testing4The Challenges and Consequences of theGrowth in Data and the User BaseBusinesses acknowledge that application performance directly affects theirbottom-line. Take the instance of a company operating within the advertisingdomain. The amount it earns is directly related to how fast its ads are gettingdisplayed on the Website.In this environment, when companies are not sure about the performanceparameters and application scalability, moving to the Cloud ensures effectivemanagement of performance requirements.Exploiting the power of the Cloud for Performance testing is gaining immensepopularity. Amazon provisioned 18 million EC2 servers last year. Also, thenumber of users subscribing to other Cloud vendors such as Microsoft hasincreased significantly.Before discussing the benefits of Cloud-based performance testing, it isimportant to examine the performance testing model, currently in use.The performance testing model involves various components such as theperformance testing tools, the system under test, etc. that are deployed ‘onpremise,’ within a local environment.One of the first steps in conducting a performance testing assignment within alocal environment is the setting up of a test lab. The lab requires infrastructure,servers, hardware and software to host the system under test, and theperformance testing tools. Setting up such a lab requires considerableinvestment and effort, and the management of overheads such as maintenancecosts. An organization setting up such a lab will need to hire administrators tomaintain and support the lab. This will further involve training costs for theresources, who will perform the support role.Also, with the substantial increase in Internet traffic, both in terms of thenumber of users and the size of the data, the critical challenge for theperformance testing team is to scale and test immediately to higherconcurrency levels.This typically calls for additional hardware, tool license extensions, and settingup the tools and other software on the new hardware, resulting in additionalcapital expenditure which may lead to a delay in getting the expectedperformance numbers.It has been observedthat the downtime of aWebsite and the latencyof even milliseconds canresult in revenue lossesand customerdissatisfaction. Thusapplicationperformance doesmatter, andorganizations doconsider it as one of thekey attributes of theirproduct strategies.
  5. 5. Harnessing the Cloud for Performance Testing5Another significant aspect of the on-premise model is that it does not measureup to the geographical distribution. An application will get hits from all cornersof the world and in order to satisfy users, the performance of the applicationneeds to be measured in peak traffic. The simulated load should mimic differentgeographies for a more accurate simulation of real usage. Furthermore, loadgenerators are an important component of performance testing and in case ofhigher volumes, need to be set up in larger numbers.The good news is that these issues can be addressed by the Cloudinfrastructure.Harnessing the Power of the CloudThe Cloud can help organizations scale for higher volume, higher transactionsand higher concurrency, quickly and affordably. It helps in addressing thelimitations indicated above and offers enterprises distinct advantages, includingthe following:Infinite computing power: The Cloud has the capability to simulate millionsand millions of users from thousands of instances. This helps in testing theapplications with production data volumes and concurrent users.Affordability: The Cloud helps in saving the upfront costs in setting up theperformance testing environment as the Controllers and load generatorscan be instantiated, as and when required.The pay-as-you-go benefit: In the case of the Cloud, organizations needonly pay for the duration the machines are being used. Thus, by planningthe tests, companies can save costs for using the hardware for a longerperiod of time. The instances can be created and configured at the time ofuse and terminated after running the tests. With the Cloud, companies donot need to worry about the licensing costs of the software, OS,performance testing tools, etc.Easy set up: The Cloud is quick and easy to set up and the test environmentcan be up and running in minutes. Many vendors provide automaticprovisioning of the load generators that helps in quickly setting up theenvironment and starting the test executions. The Cloud also offers a ready-to-use environment where no separate installation and configuration ofperformance testing tools is required. The instances are pre-configured asload generators or controllers.Ability to choose geographies: The Cloud makes it possible fororganizations to choose geographies, so that instances are created and the
  6. 6. Harnessing the Cloud for Performance Testing6load simulated from different locations. The result is that there is a muchmore accurate simulation of production usage of the application under test.Clearly, Cloud-based performance testing can be beneficial for Internet andmobile applications that have a large customer base and huge transactionvolumes.Though the Cloud offers lots of advantages, there are a few challenges thatneed to be addressed, before performance testing can become viable.The Flip Side of Cloud and Performance TestingDespite its advantages, the Cloud has its share of concern areas. Data Securityis a major issue for any performance testing activity that is performed on theCloud, since all the test scripts, test data, server response and test results havebeen moved to the Cloud. Today, however, organizations can have a privateCloud, and set up a separate tunnel for security.Another problem area is testing intranet applications on the Cloud; where thereis need to work with the client’s IT department in order to receive permission toaccess the application from the Cloud. Network latency is another critical factorthat may skew the performance numbers. It poses a challenge for intranetapplications, but can be resolved to an extent by statistically adjusting theresponse times. Also, for intranet applications, inclusion of latency in the testresults provides a realistic view of applications performance.During the performance tests, resource monitoring and diagnostic activity isdata intensive. In case of the Cloud, data transfer that happens IN and OUT ofthe Cloud is huge and may pose a challenge.Since the Cloud instances may not be available all the time, it is important tohave a pertinent test data management strategy, and a proper back-upmechanism for test scripts, test data and test results.Test script creation on the Cloud is a tough task as it involves parameterization,validation and debug runs. Ideally, the script creation should be done in a localenvironment, with the scripts being uploaded on the Cloud for full load tests.
  7. 7. Harnessing the Cloud for Performance Testing7The Tool Deployment Models forCloud-Based Performance TestingThe Cloud modelThere are some standard components of the performance testing toolarchitecture like controllers, load generators, recorders and analyzers. Thevariations in the deployment models depend on where all these componentsare placed while the performance tests are being conducted. Organizationshave to choose the appropriate deployment model based on their needs.In the case of the Cloud model, the performance testing tool is moved to theCloud and therefore, the controller, load generators and analyzer are alsodeployed on the Cloud. The Recorder however, is provided on-premise so thatusers can record the test script, debug and validate them and then further pushto the Cloud for load testing.
  8. 8. Harnessing the Cloud for Performance Testing8There is an option of having the recording component on the Cloud; howeverthat might result in extra hits to the Cloud environment during script creation. Itis recommended that the scripts are recorded locally and then moved to theCloud for scaling.This model is suitable for performance testing of Cloud-based applications orInternet applications, since these applications get hits from all parts of theworld. The scenarios can be designed in a way to simulate realistic load.For instance, in case an Internet application is deployed on the data center,where 50 percent of the traffic is coming from the east coast, 20 percent fromthe west coast and remaining from the Asia Pacific region, it should be possibleto spawn virtual users from the Cloud-based load generators instantiated fromthese regions.This represents realistic simulation as compared to the on-premise performancetesting model. It can also provide as many load generators as required, quicklyand on demand, without the need for hardware procurement, and therefore nomassive expenditure on setting up the infrastructure.Another benefit is that there is no need to install and configure the performancetesting tool and other software. There are no tool license costs involved. Theprocess becomes simplified and the organization is only charged for the periodthat it uses the Cloud infrastructure.The Hybrid modelIn this case, the components of the performance testing tool are deployedeither on the Cloud or on-premise. There are many combinations possible here,but it is feasible to take into account the most likely scenario.Here, the load generators are on the Cloud and all the other components, suchas the controller, recorder and analyzer are on-premise.This model is more suitable for organizations which have already invested inperformance test labs and have a hardware setup. These companies can use thehybrid model when they need to scale quickly.Often regular performance testing can be done on-premise, not only to get theperformance numbers, but also to determine the performance bottlenecks inthe application. However, in case there is a requirement for testing anapplication with peak volumes, high concurrencies and more transactions, theHybrid model can be used.
  9. 9. Harnessing the Cloud for Performance Testing9In the Hybrid model, the controller is deployed on-premise and enables the useof the integrated resource monitoring and diagnostic features of theperformance testing tool. This also addresses any security concerns over crucialdata, as the controller remains in-house. Since the data is available locally, it canarchive and be referred to at any suitable time.In case of resource monitoring and diagnostics on the Cloud, there will be ahuge data transfer from and into the Cloud as there is pooling of parameterslike memory, CPU, network bytes every 5 or 10 seconds.
  10. 10. Harnessing the Cloud for Performance Testing10The Impetus SolutionImpetus Technologies has developed SandStorm, an enterprise performancetesting tool, which can be deployed on the Cloud. Using this solution,organizations can reduce their performance testing costs by 60 percent.SandStorm is pre-configured on the Cloud instances and offers a ‘Ready-to-use’test environment. Moreover, the test scripts can also be created locally usingthe Recorder and automatically uploaded to the Cloud before running the tests.SandStorm supports multiple protocols ranging from the Web, mobile and e-mail like Http, Https, FLEX, Silverlight, Java applets, OMA IMPS, WAP, WV, andSMTP etc. It offers integrated resource monitoring for most of the popularapplications, Database servers and Operating Systems. It provides the UserInterface wizards for script creation and configuration.The analysis dashboards provide a summarized view of test results with drilldown capabilities to identify performance bottlenecks (visitsandstorm.impetus.com).Case Study: Performance testing using the CloudThe customer: A leading provider of workforce management solutions rangingfrom payroll, leave, schedule, among others. The company offers performancebenchmarking services to its customers wherein the performance testingactivity is conducted at the facilities of its customers.The need: The customer requested Impetus to come up with a more affordableand flexible solution for its clients.The solution: Impetus suggested using SandStorm on the Cloud. The Recordercomponent was locally deployed in the test labs to create and verify test scripts.These scripts were automatically uploaded to the Cloud for test executions. Theclient exposed the application to the Cloud by setting appropriate permissionsin the firewall. The performance testing could be easily completed in time for allthe business scenarios. The network latency was adjusted based on the networkcorrection factor identified with the help of tracert utility.The benefits: The solution helped the customer to increase customerbenchmarking activities. This in turn helped the company to achieve higherrevenues and improve customer satisfaction.
  11. 11. Harnessing the Cloud for Performance Testing11Our RecommendationSummaryIn conclusion, it can be said that moving to a Cloud-based performance testingmodel offers significant benefits to organizations. At the same time, whenmoving to this model, it is important that companies pay attention to some ofthe following recommendations:• Understand the performance goals and objectives. In the scenarios,where there is no certainty on the peak usage, it is prudent to opt for aCloud strategy.• Check to see whether the organization has uncertain demand. Theelasticity from the Cloud infrastructure provides organizations with amechanism to deal with uncertainty. Moving to the Cloud for instance,saves companies from making capital expenditure and procuringadditional hardware if the existing set-up does match peak usage, or incase the peak demand does not reach the expected capacity.• Think from the end user perspective. Cloud-based applications andInternet applications have to be accessed globally and companies need
  12. 12. Harnessing the Cloud for Performance Testing12to define proper workload patterns to simulate the realistic load. Theapplication must be tested at both peak and normal traffic levels fromall the targeted geographical locations.• Look at the type of the application that can move to the Cloud. Thisapproach is useful for applications that have high concurrent users, bigdata problems and mobile applications accessible from the Internet.• Choose appropriate tools and models for the Cloud-basedperformance testing. The models and tools should be chosen based onthe type of application, user base, non-functional requirements,scripting requirements, and test data management.• Analyze performance. The Cloud environment might be shared.Different vendors have different mechanism of providing the virtualinstances on the Cloud. In case, the underlying hardware is shared,analyze the performance and resource consumption. Any unexplainedchanges in performance may be attributed to either shared resources orthe Cloud provisioning tool.• Have continuous performance management in place. Performancetesting can be a part of continuous integration and can automaticallyexecute critical test scenarios with intermediate builds. The process mayuse automatic provisioning to generate Controller and load generatorinstances based on the peak usage of the application. These canincrease the efficiency of the entire process.About ImpetusImpetus is a Software Solutions and Services Company with deeptechnical maturity that brings you thought leadership, proactiveinnovation, and a track record of success. Our Services and Solutionsportfolio includes Carrier grade large systems, Big Data, Cloud,Enterprise Mobility, and Test and Performance Engineering.Website: www.impetus.com | Email: inquiry@impetus.com© 2013 Impetus Technologies, Inc.All rights reserved. Product andcompany names mentioned hereinmay be trademarks of theirrespective companies. May 2013