Cloud-Enabled vis-à-vis On-PremisePerformance Testing: A face-offW H I T E P A P E RAbstractIn this white paper we talk ab...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off2Table of ContentsIntroduction.............................
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off3IntroductionPerformance engineers worldwide are harness...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off4scale tests. Moreover, cloud technology enables organiz...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off5While cloud-based solutions offer several benefits for ...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off6You can also employ application performance management ...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off7for performance testing and is used by tools like JMete...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off8Due to varying network conditions, the tests might not ...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off9best practices can help you maximize the advantages, an...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off10As discussed, one of the primary concerns of users is ...
Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off11We recommend that you establish the performance testin...
Upcoming SlideShare
Loading in …5
×

Cloud-enabled Performance Testing vis-à-vis On-premise- Impetus White Paper

710 views
609 views

Published on

For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper

In this white paper we talk about how you can integrate on-premise and cloud-based options to build an effective performance testing strategy.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
710
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cloud-enabled Performance Testing vis-à-vis On-premise- Impetus White Paper

  1. 1. Cloud-Enabled vis-à-vis On-PremisePerformance Testing: A face-offW H I T E P A P E RAbstractIn this white paper we talk about how you can integrate on-premise and cloud-based options to build an effectiveperformance testing strategy.We will also discuss the available cloud-based performancetesting solutions and whether they can effectively fill the gapareas created by the on-premise approach.Finally, we share our recommended strategy which incorporatesboth on-premise and the cloud approach and presents a robustperformance testing plan.Impetus Technologies, Inc.www.impetus.com
  2. 2. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off2Table of ContentsIntroduction...........................................................................................................3Limitations of the on-premise approach ...............................................................3Benefits of cloud-based load testing .....................................................................4Challenges facing the cloud-based approach........................................................5Data security................................................................................5Testing intranet applications .......................................................5Repeatability and consistency .....................................................5Debugging and bottleneck identification ....................................5Cloud-enabled solutions........................................................................................6Using on-premise tools in the cloud............................................6Provisioning the tool from the cloud...........................................7The Hybrid approach ...................................................................8Impetus’ recommendations...................................................................................8Conclusion............................................................................................................11
  3. 3. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off3IntroductionPerformance engineers worldwide are harnessing the infinite computing powerof the cloud to test the performance of their web, mobile and cloudapplications.The immense popularity of cloud-enabled performance testing is based on thefact that it helps to overcome the limitations of a test strategy that reliesentirely on on-premise performance testing.However, before moving to cloud-based performance testing, it is important toexamine the on-premise model, so called, because all its components—including the tool, recorder, controller, system under test, etc., are deployedwithin the local environment.Limitations of the On-Premise ApproachAdministrators need to be hired to maintain and support the environment.Since setting up this environment is a costly affair, there is a possibility that youmay be requested to share this set up with other teams or for other activities,such as for a staging environment or for User Acceptance Testing.Also, it is very difficult to scale the test environment to simulate sporadic peakloads because it involves procuring additional servers, licenses etc. This mightresult in extra expenditure. However, this is mostly about the infrastructureconsiderations.The other significant aspect that we have to consider is that the on-premisemodel does not measure up to the realistic simulation of user behavior. Sinceyour application will get hits from all corners of the globe, it is important that itsperformance is measured from multiple geographies.Also, while conducting the tests from within the local environment, we ignoreaspects such as network latency, firewall behavior, DNS servers and otherimportant components in the end-to-end delivery chain. By doing this, we mightnot be testing the actual user experience.These are some of the limitations of the traditional on-premise approach. cloud-enabled performance testing can address these gap areas.At Impetus, we believe that performance testing with the cloud enables testingteams to take a big step forward in conducting more efficient and realistic, largeThe first and the very basicstep of on-premiseperformance testing aresetting up a test environment.This involves putting togethera suitable infrastructure,procuring servers, andsoftware in order to host theload generation hardware andsystem under test. Setting upthis kind of an environmentrequires considerableinvestment and effort and hasoverheads such as license andmaintenance Cost.
  4. 4. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off4scale tests. Moreover, cloud technology enables organizations to realizesignificant savings in cost and time.Benefits of Cloud-Based Load TestingThe cloud can help set up the test environment and have it running in minutes.You just do not have to worry about the licensing costs of the software,Operating System, tools, etc.Furthermore, it eliminates the need to spend on the purchase and maintenanceof the hardware and load testing tool that you would otherwise have to buy inthe on-premise model.Using the pay-as-you-go approach, you can rapidly set up the testinginfrastructure that you need, when you need it, and only for as long as you needit. From a business standpoint, the cloud lowers the Total Cost of Ownership(TCO), while increasing flexibility.Often, the web application experiences sporadic surges in traffic. Generating theload for large scale tests to mimic these unanticipated spikes in the productiontraffic, may require tens or even hundreds of machines.While purchasing and configuring these systems requires a significantinvestment in time and money, with the cloud, you can rapidly set up as manyload generating machines as you need, on-demand using automatedprovisioning of required hardware. Automated provisioning ensures that theload generators are consistent and optimized to create the desired load.Another advantage that the cloud offers is the realistic simulation of testscenarios. With the cloud, you can execute load tests that access your webapplication as your users will—from the outside of your firewall—and validateall the infrastructure components, including the firewall, DNS, networkequipment, and ISPs.These tests are significantly accurate and enable you to evaluate the real-worldeffects of third-party components, such as content delivery networks, analyticsservers, and ad servers. And since your users will not access your applicationfrom one particular geographical location, a realistic load test will requiresimulation from different geographies. This can be achieved via the cloud.Cloud-based solutions can also be accessed via browsers from multiple devices,such as your PCs, Smartphones, Tablets, etc., and therefore, you can choose torun and analyze performance tests from anywhere, anytime.
  5. 5. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off5While cloud-based solutions offer several benefits for performance testing, thecloud does have its own set of limitations. Let’s see what they are.Challenges Facing the Cloud-Based ApproachData securityWe will start with data security, which is a major concern for any performancetesting activity happening via the cloud. This is a critical challenge as the testscripts, test data, and test results are moved to the cloud. There is a need tofigure out ways by which sensitive data is not accessed via any unauthorizedsource.Testing intranet applicationsAnother problem area is testing intranet applications on the cloud. Performancetesting service providers actually need to work with the client’s IT department inorder to receive permission to access the application from the cloud. Thisinvolves exposing the intranet servers to the external world and requiresrelevant security mechanisms in place, to avoid any unauthorized access.Repeatability and consistencyThe other challenges are repeatability and consistency of the test results. Oftenin performance testing, you need to precisely measure the effect of changesmade to the application code or configurations. For example, you may need todetermine any improvement that results from resolving a specific defect, orevaluate performance for a range of cache sizes and other related settings.With cloud load testing, such precise measurements are difficult because of thevariations in Internet traffic and bandwidth availability. Such variations canmake it almost impossible to duplicate uniform test conditions.On the other hand, on-premise testing can achieve consistent and repeatableresults by isolating effects that are due to the application or infrastructure.Debugging and bottleneck identificationDebugging and bottleneck identification is another major challenge when thetests are conducted from the cloud. When load testing uncovers a problem, thenext step is identifying which layer in the application chain is causing theproblem. You can use monitors to check the performance metrics such as hits,average response time per request, and average bandwidth for each layer in thechain.
  6. 6. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off6You can also employ application performance management (APM) solutions toidentify bottlenecks in the code. These techniques work reasonably well whenthe environment is controlled and free from external factors.Let us now move on to the next problem area—Test Assets Management.Since the cloud instances may not be available all the time, it is important tohave a pertinent test data management strategy in place, as well. A proper backup mechanism for test scripts, test data and test results has to be created, sothat they are available for reference, at a later stage.After going through the benefits and challenges of using the cloud forperformance testing activities, let us look at the solutions available in themarket today that can help in optimally leveraging the cloud.Cloud-Enabled SolutionsThese solutions can be classified into three major categories.The first and simplest solution is to move the current on-premise tool to thecloud, as it is. A few companies also offer automated Ccoud provisioning portalsfor load generation, as well as executing and controlling tests.The first two categories have a few limitations and hence a third category calledthe ‘Hybrid approach’ is in place. This involves using on-premise tools togenerate on-demand load, using the cloud. It supports high concurrency testingwhile still providing the benefits of a controlled environment and security.Impetus recommends a two-phase approach, which we believe brings the bestof on-premise and cloud options and can help you to build a successfulperformance testing strategy.Let us examine each of these strategies in detail:Using on-premise tools in the cloudAs we know, there are some standard components ofperformance testing tools such as controllers, load generators,recorders and analyzers.The variation in the deployment models depends on where allthese components are placed when we conduct the tests. Youneed to choose the appropriate deployment model based on yourneeds.In the case of the first model, all components of the tool aremoved to the cloud. This is the simplest way to leverage the cloudThis model is suitable to test cloud-based applications or Internetapplications, since these get hitsfrom all parts of the world. Thescenarios can be designed in amanner that simulates a realisticload across multiple geographies.
  7. 7. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off7for performance testing and is used by tools like JMeter, Grinder and LoadRunner.This solution helps in generating a realistic simulation, unlike the on-premisemodel. You also have the option of instantiating as many load generators as yourequire, quickly and on-demand. Since there is no hardware procurementinvolved, you are unlike to incur massive expenditure on setting up theinfrastructure.Another benefit of this approach is that there is no learning curve involved. Theteams are already familiar with the tool and can start testing quickly. Also, thereare no license costs involved. The process becomes simplified and you are onlycharged for the period you use the cloud infrastructure.However, there are severallimitations in this model. Itrequires manual configuration ofinstances and knowledge aboutcloud APIs. The instances can onlybe accessed via RDP, Putty or SSH,which involve physical connectionto the machines. Also, the testdata resides in the cloud and afterthe instance is terminated can belost. The user has therefore tocome up with an appropriate dataarchival strategy beforeterminating the instances.Provisioning the tool from thecloudLet’s now talk about another popular approach, which is provisioning the toolfrom the cloud. This involves using the tool provisioned on the cloud for loadgeneration, execution and test management. All the components of a loadtesting tool are deployed on the cloud and the user can access the solution fromthe browser. This provides an easy interface that can be accessed fromanywhere, at anytime.Though this approach is very popular, it has a few limitations that you need toknow about. Since the tool is accessed via the Internet, data security is one ofthe major concerns. All the test artifacts and results are stored on the cloud,thereby, presenting a challenge to data confidentiality. Also, since the tests arerun over the Internet, appropriate firewall access needs to be provided for theintranet applications.Impetus SandStorm enables you toprovision the tool from the cloud.One of the major benefits of thisapproach is on-demand loadgeneration, which means the usercan scale its load tests at any giventime. Pre-created instances help inquick scaling of the load tests. Thisprocess is suitable for mobile andcloud-based applications that havea large user base, and scalesquickly.
  8. 8. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off8Due to varying network conditions, the tests might not be repeatable.It is also difficult to identify performance bottlenecks in such cases, as thenumber of variables is relatively high. This approach is well suitedfor measuring the end user experience and exercising the entiredelivery chain including firewalls, ISP, CDN, routers etc. in atypical application deployment.Now, let us talk about the Hybrid approach that tries to addressthe limitations of the earlier two approaches.The Hybrid approachIn this case, the components of the performance testing tool aredeployed either on the cloud or on-premise. There are manycombinations possible here, but let us take up the most likelyscenario to understand this better.This is because regular testing can be done on premise not only toget the performance numbers, but also to determine the bottlenecks in theapplication. In case there is need to test an application with peak volumes, highconcurrencies and more transactions, the Hybrid model can be used.Impetus’ SandStorm and a few other tools offer this option. The controller isdeployed on-premise in this model and enables you to use the integratedresource monitoring and diagnostic features of the tool. This also helps addressany security concerns over the crucial data as the controller remains in-house.Since the data is available locally, it can be archived and referred to at anysuitable time.This model too suffers from a couple of limitations related to remote access andparallel test executions. Since, the controller is installed within the localnetwork; the tests can only be executed via physical access to these machines.Impetus’ RecommendationsWe will now talk about some of the work we have done and the best practiceswe have deployed successfully across multiple customer engagements. TheseWe have the load generators onthe cloud and all the othercomponents, such as, thecontroller, recorder and analyzeron-premise. This model is moresuitable for organizations whichhave already invested inperformance test labs and have ahardware set up. Theseorganizations can use the Hybridmodel when they need to scalequickly.
  9. 9. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off9best practices can help you maximize the advantages, and minimize thechallenges of load testing with the cloud.As we discussed earlier, while the on-premise approach has numerouslimitations, it cannot be replaced completely by the cloud. And while cloudoffers scalability along with cost-effectiveness, it also has some pertinentchallenges associated with it. Therefore, let us examine the fourth approachthat integrates the cloud and on-premise solutions for a successful and effectiveperformance testing strategy.The on-premise model is suitable for initial performance tests and debuggingand forms the first stage of our process. You can use the local environment tocreate test scripts, prepare test data, and validate the test scripts for runtimedata changes. Then, can you conduct internal tests with a medium load toquickly identify and resolve preliminary issues.The source of any issue identified in the first stage is clearly within the firewall(because no other external systems are involved in the test, as of now).Therefore, it is easier to pinpoint and fix internal problems when they are notbeing compounded by other issues that will originate outside the firewall.With this two-stage process, you do not have to wait for the application to bedeployed and accessible from the Internet to test it. You can test the applicationinternally, and earlier in the application lifecycle, when the defects are easierand less expensive to fix. With internal testing, you have much more controlover the environment, so you can precisely measure the effect that a code or aconfiguration change might have on the application performance.The tests are repeatable, and therefore, the tuning/optimization changes can bequickly validated. This approach also helps in creating scripts locally, and retainsthem for scaling at a later stage, using the cloud. After the performance hasbeen validated internally, you can proceed to the second stage, which is cloud-based load testing.The cloud can be used to test large scale, a peak load that validates the entiredelivery chain of the application. You can compare the results of the same testscenario run internally and from the cloud, to get a better handle on how theapplication server and network infrastructure contribute to the overall responsetimes.cloud testing is based on pay-per-use model, and since you have already testedthe application internally on your existing hardware setup, you have an optionto reduce the amount of cloud testing and save on costs. You can also detectpotential network issues at the cloud provider level. And having gone throughthis model, we will talk about the best practices associated with using cloud-based performance testing solutions.
  10. 10. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off10As discussed, one of the primary concerns of users is data security. Werecommend encrypting the communication between your controller and loadgenerators. This will help secure the data sent to the load generators during thetest (including account information), as well as the data that is retrieved(including error messages). If possible, use SSL to secure the communicationbetween the browser and the test server. Finally, ensure that your loadgenerators are secured with their own firewalls to protect them from outsidethreats.To ensure that your load generator machines in the cloud are capable ofgenerating larger loads, we would recommend tuning the system to support thecreation of a high number of sockets and threads per process. Additionally,allocating an appropriate heap size for Java-based load generators will help insuch scenarios. The default settings for a typical machine allow all programs toshare resources fairly. In the case of load generators, the machine is dedicatedto a single task, so you can improve the performance by allocating a significantlylarger share of the available resources to the load generation tasks.Automated provisioning can also save time and efforts to scale the testenvironment to support high concurrency and transaction volumes. It alsomakes sure that the load generators are consistent and optimally tuned as theyare generated from a single image.We recommend using an integrated solution for on-premise and cloud. If thesolutions are interoperable, it reduces the learning curve for the team. The testscripts, scenarios can be re-used internally as well as on the cloud.Having covered the four available cloud-based performance testing solutions,we would like to point out that Impetus’ two-phased approach appears to meetall the criteria of a robust performance testing strategy.Almost all other approaches, other than cloud provisioning, need toolfamiliarity. Even the cloud provisioning option does not allow reuse of thescripts created locally for large scale testing from the cloud.A major advantage that the cloud provisioning and Impetus approaches offer isthe ability to run multiple tests in parallel. This is very critical for largeorganizations where multiple parallel releases are happening for differentapplications. Another advantage is that they both provide browser-basedtesting to simulate the end user experience. This is very important, owing to therapid changes taking place in client side technologies like HTML5, FLEX, etc.Each of these four solutions that we have spoken about have their own meritsand demerits. The selection of specific tools depends on factors such as cost-effectiveness, type of application, performance objectives, etc.
  11. 11. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off11We recommend that you establish the performance testing goals and thendecide on the preferred solution.ConclusionAt the end it can be said that organizations that use only lab testing or onlycloud testing are at a disadvantage. This also holds good for the companies thatuse different cloud-based solutions for these activities. We at Impetus believethat a solution that integrates the on-premise and cloud-based approaches willbe the most beneficial.We have put together a quick check list to help you decide when to choose anon-premise option vis-a-vis the cloud. The on-premise environment is a highlycontrolled one, hence it is well suited to preliminary, low-scale testing. You canstart testing early in the development cycle when the code is not yet ready forhigh concurrency testing.This helps in resolving performance issues early. Also, since the environment isconsistent, it ensures repeatability of the tests. Any code change and tuning canbe quickly certified for performance improvements in case of on-premise.The cloud, meanwhile, offers elasticity on demand i.e. large scale tests with highconcurrency can be executed anytime. There is no need to procure additionalhardware, licenses etc. This is well suited for applications that can scale quickly,such as mobile and cloud applications. From the cloud, we exercise theapplication under test from an external world, thus bringing in the CDN, ISP androuters etc. in the picture. This is very similar to how an end user will be usingthe system.About ImpetusImpetus is a Software Solutions and Services Company with deeptechnical maturity that brings you thought leadership, proactiveinnovation, and a track record of success. Our Services and Solutionsportfolio includes Carrier grade large systems, Big Data, Cloud,Enterprise Mobility, and Test and Performance Engineering.Website: www.impetus.com | Email: inquiry@impetus.com© 2013 Impetus Technologies, Inc.All rights reserved. Product andcompany names mentioned hereinmay be trademarks of theirrespective companies. May 2013

×