Load Testing as a Part of
         the Software Development
                   Cycle
                                     ...
Agenda
         Performance Testing

         “Record and Playback“

         Alternatives




Copyright © 2004, Hyperion ...
Performance Testing
         Testing multi-user applications for
         performance is a must today

         You never ...
Typical Questions
         What would be response times for 100
         concurrent users?
            – Performance / loa...
Hyperion Solutions
 Presentation is based on Hyperion performance
 team experience

 Hyperion Solutions is a vendor of Bus...
Performance Testing at
                         Hyperion
        Centralized Performance Engineering Group
        was cre...
All Stages of Software Life
                     Cycle
         Technology evaluation
         Prototypes
         Compone...
Load Testing Process
         Define (design) what you want to test

         Fill it with test data

         Create work...
Workload Development
     Workload development is the most non-trivial
     part of work according to our experience

    ...
Workload
               A workload should reproduce the typical
               stress on a system

               A good w...
Agenda
         Performance Testing

         “Record and Playback“

         Alternatives




Copyright © 2004, Hyperion ...
"Record and Playback"

       Virtual users: record communication
       between two tiers and then playback an
       aut...
Virtual User Simulation
                                                                                 Virtual
         ...
Load Testing Tools
  List of supported features differs significantly
  from tool to tool

  Universal powerful tools:
   ...
Features of Universal Tools
        Ability to record scripts automatically for
        different protocols

        A num...
Features of Universal Tools
         Advanced script language for workload
         development

         Ability to call ...
Load Testing Tools
         A lot of specialized tools
            – www.softwareqatest.com/qatweb1.html
            – tes...
Open Source
         OpenSTA
            – HTTP/S
            – www.opensta.org


         Apache JMeter
            – Web...
Other Ways
         Appliances
            – For example, Spirent Avalanche
            – can be useful for simulation a b...
Problems
         "Record and playback" approach often
         doesn't work for testing components

         Each load te...
Hyperion Enterprise
           Proven financial consolidation, reporting,
                         and analysis


        ...
Hyperion Financial Management

                     Win32                                         DCOM                    ...
Hyperion Reports

                     Win32                                        Java RMI                              ...
Agenda
         Performance Testing

         “Record and Playback“

         Alternatives




Copyright © 2004, Hyperion ...
GUI Users
         Functional / regression testing tools
            – WinRunner, QuickTest Pro, Rational Robot, etc.


  ...
GUI Users

          Requires a real machine for each
                         user

            – Mercury can use one Win...
Custom Test Harness
         Special program to generate workload

         Requires access to the API or source code

   ...
Advantages
         Doesn’t require any special tool

         Starting version could be quickly created by
         a pro...
Disadvantages
         Efforts to update and maintain harness can
         increase drastically

         When you have nu...
Custom Load Generation
         Mixed approach
            – Lightweight custom client stubs to work with an
             ...
Custom Load Generation
       Client PC                                                  Load generation PC


        Appl...
Implementation
         We did it for LoadRunner and Rational Test

         Standard external DLL in C/C++

         API ...
Advantages
        Eliminates dependency on supporting
        specific protocols

        Leverages all the features of L...
Considerations
               Requires access to API or source code

               Requires programming

               M...
Recording vs. API
   RMI recording
  _integer =
    _ireportserver.executeJob(_designjobobject);
  _ireportserver.getStatu...
More Considerations
         Requires a load test tool license for the
         necessary number of virtual users

       ...
If Difficult to Parameterize…
               Recording and parameterization of a
               script could be time-consu...
Example: Essbase Query
         Multi-Dimensional Database

         C API
            – Used by many applications and mid...
Winsock Script
send buf22 26165
  "xffx00xf0a"
  "x00x00x00x00x01x00x00x00x01x00x03x00"
  "dx00bx00"
  "y'<Handle1>x00"
  ...
Script Using External DLL
lr_load_dll("c:templr_msas2k.dll");
pCTX = Init_Context();
hr = Connect(pCTX, "ess01", "user001"...
Summary
     Performance testing is a must today for multi-
     user applications

     Performance testing is not one-ti...
To Learn More
         My collection of link (just started)
         www.alexanderpodelko.com
         www.perftestplus.co...
Upcoming SlideShare
Loading in...5
×

Load Testing as a Part of Load Testing as a Part of the ...

1,563

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,563
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Load Testing as a Part of Load Testing as a Part of the ...

  1. 1. Load Testing as a Part of the Software Development Cycle PHILACMG November 19, 2004 Alexander Podelko Hyperion Solutions Alexander_Podelko@hyperion.com Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 1
  2. 2. Agenda Performance Testing “Record and Playback“ Alternatives Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 2
  3. 3. Performance Testing Testing multi-user applications for performance is a must today You never know how an application will work with 1,000 users until you test What you need to do significantly depends on what your business is Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 3
  4. 4. Typical Questions What would be response times for 100 concurrent users? – Performance / load testing What happens under excessive load? – Stress testing What hardware do we need for 100 users? – Capacity planning Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 4
  5. 5. Hyperion Solutions Presentation is based on Hyperion performance team experience Hyperion Solutions is a vendor of Business Performance Management software – Revenues of $622 millions in fiscal 2004 – Packaged applications and tools Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 5
  6. 6. Performance Testing at Hyperion Centralized Performance Engineering Group was created in 1997 Lab environment & customer sites Numerous products and configurations Now each development group makes it own performance testing Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 6
  7. 7. All Stages of Software Life Cycle Technology evaluation Prototypes Components Pre-release / release Benchmarking Pre-sales POC Before going live Performance issues in production Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 7
  8. 8. Load Testing Process Define (design) what you want to test Fill it with test data Create workload Apply workload to the system Analyze results Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 8
  9. 9. Workload Development Workload development is the most non-trivial part of work according to our experience We need to create meaningful and realistic workloads in a timely manner – Usually in the development cycle timeframe – Different for each product / interface Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 9
  10. 10. Workload A workload should reproduce the typical stress on a system A good workload for performance testing should be: – Measurable – Reproducible – Static – Representative Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 10
  11. 11. Agenda Performance Testing “Record and Playback“ Alternatives Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 11
  12. 12. "Record and Playback" Virtual users: record communication between two tiers and then playback an automatically created script We successfully used this approach in most project since 1997 We used two load testing tools: Mercury LoadRunner and Rational Test (Performance Studio, preVue) Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 12
  13. 13. Virtual User Simulation Virtual User Controller Clients Results Repository Client Transaction Transaction Prototype Network Web App Web App Server Server Server Server Database Database Server Server Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 13
  14. 14. Load Testing Tools List of supported features differs significantly from tool to tool Universal powerful tools: – Segue SilkPerformer (www.segue.com) – Rational Test (www.rational.com) – Compuware QA Load (www.compuware.com) – Mercury LoadRunner (www.mercury.com) Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 14
  15. 15. Features of Universal Tools Ability to record scripts automatically for different protocols A number of simulated users limited mainly by available hardware Centralized test management and result analysis Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 15
  16. 16. Features of Universal Tools Advanced script language for workload development Ability to call external functions Ability to simulate GUI users as well as virtual users Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 16
  17. 17. Load Testing Tools A lot of specialized tools – www.softwareqatest.com/qatweb1.html – testingfaqs.org/t-load.html Empirix (Web) – Same scripts for functional and performance testing Microsoft Application Center Test (ACT) comes with Visual Studio .Net Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 17
  18. 18. Open Source OpenSTA – HTTP/S – www.opensta.org Apache JMeter – Web, JDBC – jakarta.apache.org/jmeter www.opensourcetesting.org/performance.php – List of 18 open source tools Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 18
  19. 19. Other Ways Appliances – For example, Spirent Avalanche – can be useful for simulation a big number of simple Web users – Limited parameterization Outsourcing Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 19
  20. 20. Problems "Record and playback" approach often doesn't work for testing components Each load testing tool support a limited number of technologies (protocols) We had several problems back in 1999 Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 20
  21. 21. Hyperion Enterprise Proven financial consolidation, reporting, and analysis Enterprise SMB Protocol Win32 Client Data Files Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 21
  22. 22. Hyperion Financial Management Win32 DCOM Application Client Server Web Client HTTP Web Server ASP Repository Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 22
  23. 23. Hyperion Reports Win32 Java RMI Reports Client Server Web Client HTTP Web Data Source Application Repository Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 23
  24. 24. Agenda Performance Testing “Record and Playback“ Alternatives Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 24
  25. 25. GUI Users Functional / regression testing tools – WinRunner, QuickTest Pro, Rational Robot, etc. Record and playback communication between user and client GUI Don't care about communication protocols / internals Accurate data (real client, end-to-end) Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 25
  26. 26. GUI Users Requires a real machine for each user – Mercury can use one Windows Terminal session per user, so running several GUI users on the box – Another workaround from Mercury is using low- level graphical Citrix protocol Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 26
  27. 27. Custom Test Harness Special program to generate workload Requires access to the API or source code Requires programming Could be cost effective solution in some simple cases Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 27
  28. 28. Advantages Doesn’t require any special tool Starting version could be quickly created by a programmer familiar with API Should work if API works You don't care what protocol is used for communication Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 28
  29. 29. Disadvantages Efforts to update and maintain harness can increase drastically When you have numerous products you really need to create something like a commercial load testing tool Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 29
  30. 30. Custom Load Generation Mixed approach – Lightweight custom client stubs to work with an application – Commercial load testing tool to manage these stubs and analyze results Implementation depends on the particular tool – We used Rational Test and Mercury LoadRunner Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 30
  31. 31. Custom Load Generation Client PC Load generation PC Application GUI modules Load test tool Application Application comm. communication modules modules (client stub) Application or database Application or database server server Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 31
  32. 32. Implementation We did it for LoadRunner and Rational Test Standard external DLL in C/C++ API calls directly inserted into scripts – for scripts in Java, for example Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 32
  33. 33. Advantages Eliminates dependency on supporting specific protocols Leverages all the features of LoadRunner and allows using it as a test harness Sometimes simplifies work with difficult to parameterize protocols Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 33
  34. 34. Considerations Requires access to API or source code Requires programming Minimal transaction that could be measured is an external function Requires understanding of internals Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 34
  35. 35. Recording vs. API RMI recording _integer = _ireportserver.executeJob(_designjobobject); _ireportserver.getStatus(new Integer(3)); _ireportserver.getStatus(new Integer(3)); _ireportserver.getStatus(new Integer(3)); _iinstance = _ireportserver.getInstance(new Integer(3)); Real code joID = poReportServer.executeJob(djo); bStatus = true; while (bStatus) { bStatus = poReportServer.getStatus (joID); Thread.sleep(300); } poReportServer.getInstance(joID); Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 35
  36. 36. More Considerations Requires a load test tool license for the necessary number of virtual users Environment should be set on all agents Usually requires more resources on agent machines Results should be cautiously interpreted Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 36
  37. 37. If Difficult to Parameterize… Recording and parameterization of a script could be time-consuming “Custom load generation" approach sometimes can be a better choice Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 37
  38. 38. Example: Essbase Query Multi-Dimensional Database C API – Used by many applications and middleware – Winsock scripts Quite difficult to parameterize and verify External DLL was made for major functions Copyright © 2004, Hyperion Solutions Corp. All rights reserved. *All brands and trademarks are the property of their owners Slide 38
  39. 39. Winsock Script send buf22 26165 "xffx00xf0a" "x00x00x00x00x01x00x00x00x01x00x03x00" "dx00bx00" "y'<Handle1>x00" "brx00x06x00fx00x1bex00x00rx00xd6aRN" "x1ax00x06x00x00x00x00x00x00x00x00x00b" "x00x00x00xe7x00x00x01x00x03x00x04x00" "x10x00xccx04x05x00x04x00x80xd0x05x00t" "x00x02x00x02x00bx00<x00x04" "FY04aWorkingtYearTotaltELEMENT-FtProduct-P" "x10<entity>tx00x02x00" … Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 39
  40. 40. Script Using External DLL lr_load_dll("c:templr_msas2k.dll"); pCTX = Init_Context(); hr = Connect(pCTX, "ess01", "user001","password"); … lr_start_transaction("Mdx_q1"); sprintf(report, "SELECT %s.children on columns, %s.children on rows FROM Shipment WHERE ([Measures].[Qty Shipped], %s, %s)", lr_eval_string("{day}"), lr_eval_string("{product}"), lr_eval_string("{customer}"), lr_eval_string("{shipper}")); hr = RunQuery(pCTX, report); lr_end_transaction("Mdx_q1",LR_AUTO); Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 40
  41. 41. Summary Performance testing is a must today for multi- user applications Performance testing is not one-time action, it is continuous efforts during all software lifecycle No universal approach – you need to find your own way Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 41
  42. 42. To Learn More My collection of link (just started) www.alexanderpodelko.com www.perftestplus.com www.performancetester.com www.loadtester.com www.stickyminds.com Questions ? Copyright © 2004, Hyperion Solutions Corp. All rights reserved. Slide 42
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×