Sponsored by
Load Testing:
See a Bigger Picture
Alexander Podelko
alex.podelko@oracle.com
www.alexanderpodelko.com
@apodelko
April 1, 2...
Speaker Bio
• Have specialized in performance for the last 17 years
• Currently performance testing and optimization of
Hy...
In This Session, You’ll Learn…
• Load testing and its role in performance risk
mitigation
• Different approaches: load gen...
Load Testing: Terminology
Applying multi-user synthetic load to the system
• Load testing
• Performance testing
• Stress t...
The Stereotype
• Load / PerformanceTesting is:
– Last moment before deployment
– Last step in the waterfall process
– Prot...
Load Testing
• It is only one very specific kind of load testing
– Most popular due to easier integration in corporate SDL...
Performance Risk Mitigation
• Single-user performance engineering
– Profiling, WPO, single-user performance
• Software Per...
But all of them
don’t replace
load testing
Load testing
complements them in
several important ways
9
What Load Testing Adds
• Verification that the system handles the load
• Verification of multi-user performance
• Performa...
Agenda
• Load testing and its role in performance risk
mitigation
• Different approaches: load generation
• Different appr...
Load Testing
Process
Collect Requirements
Create Test Assets
Define Load
Run Tests
Analyze Results
Done
Modify System
Goal...
Load Generation
• Create test assets – run test
• A must step in load testing
• ‘Test assets’ - usually scripts or program...
Record and Playback:
Protocol Level
• Virtual users: record communication between
two tiers and then playback an automatic...
Record and Playback:
Protocol Level
Load Testing Tool
Virtual Users
ServerLoad Generator
Application
Network
15
Issues
• Usually doesn't work for testing components
• Each tool support a limited number of
technologies (protocols)
• So...
Recalling Challenges
• 199x – SMB (Server Message Block), DCOM,
Java RMI
• 200x – Applets and ActiveX controls
• 201x – Ri...
Not so Simple: Example
• Back-end calculation (Financial consolidation)
– Long time, shows progress bar
– Polling back-end...
Recorded Script
web_custom_request("XMLDataGrid.asp_7","URL={URL}/Data/XML
DataGrid.asp?Action=EXECUTE&TaskID=1024&RowStar...
Working Script
web_custom_request("XMLDataGrid.asp_7","URL={URL}/Data/XML
DataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart...
Alternatives
• Manual
• Real Users
• Record and Playback, UI Level
• Programming
• Mixed
21
Manual
• Not an option for a large number of users
• Always variation in human input times
• Can be a good option to simul...
Real Users
• Full load (testing in production)
• Partial load (A/B testing, canary testing)
• You trade in the need to gen...
Considerations
• Make sense for the following conditions
– Potential issues have minimal impact on user
satisfaction and c...
Record and Playback: UI Level
• Functional / regression testing tools
• Record and playback communication between
user and...
History of the Approach
• Traditional tools, fat clients
– Require a separate machine (or a terminal session) per user
• L...
Record and Playback: UI Level
Load Testing Tool
Virtual
Users
ServerLoad Generator
Application
Network
Browsers
27
Challenges
• Scalability
– Still require more resources
• Supported technologies
• Timing accuracy
• Playback accuracy
– F...
Programming: Custom Test Harness
• Special program to generate workload
• Requires access to the API or source code
• Requ...
Advantages
• Doesn’t require any special tool
• Starting version could be quickly created by
a programmer familiar with AP...
Disadvantages
• Efforts to update and maintain harness
increase drastically as you need to add
features
• When you have nu...
Mixed Approach
• Programming using load testing tools
– Program a script instead of recording
– Run it using a load testin...
Advantages
• Eliminates dependency on supporting
specific protocols
• Leverages all the features of the load testing
tool ...
Considerations
• Requires access to API or source code
• Requires programming
• Minimal transaction that could be
measured...
More Considerations
• Requires a load test tool license for the
necessary number of users
• Environment may need to be set...
Example 1: Essbase Query
• Multi-Dimensional Database
• C API
– Used by many applications and middleware
– Winsock scripts...
Winsock Script
lrs_create_socket("socket0", "TCP", "LocalHost=0",
"RemoteHost=ess001.hyperion.com:1423", LrsLastArg);
lrs_...
Winsock Script
send buf22 26165
"xffx00xf0a"
"x00x00x00x00x01x00x00x00x01x00x03x00"
"dx00bx00"
"y'<Handle1>x00"
"brx00x06x...
Script Using External DLL
lr_load_dll("c:templr_msas2k.dll");
pCTX = Init_Context();
hr = Connect(pCTX, "ess01", "user001"...
Example 2: EDS
• Essbase Deployment Services
• Middleware, no GUI interface
• Functional test scripts in Java
• Solution -...
EDS Java Script
import lrapi.lr;
…
public int action() {
lr.start_transaction("01_Create_API_instance");
ess = IEssbase.Ho...
Agenda
• Load testing and its role in performance risk
mitigation
• Different approaches: load generation
• Different appr...
Environments
• Lab vs. Service (SaaS) vs. Cloud (IaaS)
– For both the SUT and load testing tool
• Test vs. Production
• No...
Scenarios
• System validation for high load
– Outside load (service or cloud), production system
– Wider scope, lower repe...
Agenda
• Load testing and its role in performance risk
mitigation
• Different approaches: load generation
• Different appr...
Load Testing Tools
• Differ drastically
– Supported approaches / protocols
– Scripting / extendibility
– Supported environ...
Load Testing Tools
• There is no best tool – it depends on your
needs
• Almost every tool will work for a simple web
site
...
Summary
• Load testing is an important way of
performance risk mitigation / part of the
performance engineering process
– ...
Any questions?
49
Alexander Podelko
alex.podelko@oracle.com
www.alexanderpodelko.com
@apodelko
Load Testing: See a Bigger Picture, ALM Forum, 2014
Upcoming SlideShare
Loading in...5
×

Load Testing: See a Bigger Picture, ALM Forum, 2014

17,477

Published on

Load Testing: See a Bigger Picture. Different approaches to load / performance tetsing.

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total Views
17,477
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Load Testing: See a Bigger Picture, ALM Forum, 2014

  1. 1. Sponsored by
  2. 2. Load Testing: See a Bigger Picture Alexander Podelko alex.podelko@oracle.com www.alexanderpodelko.com @apodelko April 1, 2014
  3. 3. Speaker Bio • Have specialized in performance for the last 17 years • Currently performance testing and optimization of Hyperion products at Oracle • Board director at CMG (http://cmg.org), organization of performance and capacity professionals – Next conference November 3-6, 2014 in Atlanta, GA Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of my current or previous employers. All brands and trademarks mentioned are the property of their owners. 3
  4. 4. In This Session, You’ll Learn… • Load testing and its role in performance risk mitigation • Different approaches: load generation • Different approaches: environments • Load testing tools 4
  5. 5. Load Testing: Terminology Applying multi-user synthetic load to the system • Load testing • Performance testing • Stress testing • Scalability testing • Volume testing • Reliability testing • Concurrency testing • Endurance testing • Longevity testing • Soak testing • Stability testing 5
  6. 6. The Stereotype • Load / PerformanceTesting is: – Last moment before deployment – Last step in the waterfall process – Protocol Level Record-and-Playback – Large corporations – Expensive tools requiring special skills – Lab environment – Scale-down environment – … 6
  7. 7. Load Testing • It is only one very specific kind of load testing – Most popular due to easier integration in corporate SDLC • But load testing in no way limited to this one Technology evaluation Infrastructure evaluation Prototypes / POC Component / unit What/if Performance troubleshooting Performance optimization Benchmarking 7
  8. 8. Performance Risk Mitigation • Single-user performance engineering – Profiling, WPO, single-user performance • Software Performance Engineering – Modeling, Performance Patterns • Instrumentation / APM / Monitoring – Production system insights • Capacity Planning/Management – ResourcesAllocation • Continuous Integration / Deployment – Ability to deploy and remove changes quickly 8
  9. 9. But all of them don’t replace load testing Load testing complements them in several important ways 9
  10. 10. What Load Testing Adds • Verification that the system handles the load • Verification of multi-user performance • Performance optimization – Exactly the same load • Debugging/verification of multi-user issues • Testing self-regulation functionality – Such as auto-scaling or changing the level of service depending on load 10
  11. 11. Agenda • Load testing and its role in performance risk mitigation • Different approaches: load generation • Different approaches: environments • Load testing tools 11
  12. 12. Load Testing Process Collect Requirements Create Test Assets Define Load Run Tests Analyze Results Done Modify System Goals are met Goals are not met 12
  13. 13. Load Generation • Create test assets – run test • A must step in load testing • ‘Test assets’ - usually scripts or programs in load testing • Time constraints can make it very challenging – Different for each product / interface 13
  14. 14. Record and Playback: Protocol Level • Virtual users: record communication between two tiers and then playback an automatically created script • Usually after proper correlation / parameterization 14
  15. 15. Record and Playback: Protocol Level Load Testing Tool Virtual Users ServerLoad Generator Application Network 15
  16. 16. Issues • Usually doesn't work for testing components • Each tool support a limited number of technologies (protocols) • Some technologies are very time-consuming • Workload validity in case of sophisticated logic on the client side is not guaranteed 16
  17. 17. Recalling Challenges • 199x – SMB (Server Message Block), DCOM, Java RMI • 200x – Applets and ActiveX controls • 201x – Rich InternetApplications It was just a short period of time whenWeb sites were simple – it was rather an exception 17
  18. 18. Not so Simple: Example • Back-end calculation (Financial consolidation) – Long time, shows progress bar – Polling back-end – Explicit loop is needed to work properly 18
  19. 19. Recorded Script web_custom_request("XMLDataGrid.asp_7","URL={URL}/Data/XML DataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColS tart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript", LAST); web_custom_request("XMLDataGrid.asp_8","URL={URL}/Data/XML DataGrid.asp?Action=GETCONSOLSTATUS", LAST); web_custom_request("XMLDataGrid.asp_9","URL={URL}/Data/XML DataGrid.asp?Action=GETCONSOLSTATUS", LAST); web_custom_request("XMLDataGrid.asp_9","URL={URL}/Data/XML DataGrid.asp?Action=GETCONSOLSTATUS", LAST); 19
  20. 20. Working Script web_custom_request("XMLDataGrid.asp_7","URL={URL}/Data/XML DataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&Col Start=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript", LAST); do { sleep(3000); web_reg_find("Text=1","SaveCount=abc_count",LAST); web_custom_request("XMLDataGrid.asp_8","URL={URL}/Data/X MLDataGrid.asp?Action=GETCONSOLSTATUS", LAST); } while (strcmp(lr_eval_string("{abc_count}"),"1")==0); 20
  21. 21. Alternatives • Manual • Real Users • Record and Playback, UI Level • Programming • Mixed 21
  22. 22. Manual • Not an option for a large number of users • Always variation in human input times • Can be a good option to simulate quickly a few users • Can be used with other methods to verify correctness 22
  23. 23. Real Users • Full load (testing in production) • Partial load (A/B testing, canary testing) • You trade in the need to generate and validate workload for a possibility of performance issues and load variability. 23
  24. 24. Considerations • Make sense for the following conditions – Potential issues have minimal impact on user satisfaction and company image – Easy rollback of the changes – Homogenous workload and a way to control it – Fully parallel and scalable architecture 24
  25. 25. Record and Playback: UI Level • Functional / regression testing tools • Record and playback communication between user and clientGUI • Don't care about communication protocols / internals • Accurate data (real client, end-to-end) 25
  26. 26. History of the Approach • Traditional tools, fat clients – Require a separate machine (or a terminal session) per user • Low-level graphical protocols – Citrix, Remote Desktop • Web tools, browser – Require a separate browser instance • Web tools, light-weight browser – Require a separate light-weight browser instance – For example, HtmlUnit or PhantomJS 26
  27. 27. Record and Playback: UI Level Load Testing Tool Virtual Users ServerLoad Generator Application Network Browsers 27
  28. 28. Challenges • Scalability – Still require more resources • Supported technologies • Timing accuracy • Playback accuracy – For example, for HtmlUnit 28
  29. 29. Programming: Custom Test Harness • Special program to generate workload • Requires access to the API or source code • Requires programming • Could be cost effective solution in some simple cases 29
  30. 30. Advantages • Doesn’t require any special tool • Starting version could be quickly created by a programmer familiar with API • Should work if API works • You don't care what protocol is used for communication 30
  31. 31. Disadvantages • Efforts to update and maintain harness increase drastically as you need to add features • When you have numerous products you really need to create something like a commercial load testing tool 31
  32. 32. Mixed Approach • Programming using load testing tools – Program a script instead of recording – Run it using a load testing tool • Implementation depends on the particular tool – Scripting support – Languages support – API calls (may require external dll / client stubs) 32
  33. 33. Advantages • Eliminates dependency on supporting specific protocols • Leverages all the features of the load testing tool and allows using it as a test harness • Sometimes simplifies work with difficult to parameterize protocols 33
  34. 34. Considerations • Requires access to API or source code • Requires programming • Minimal transaction that could be measured is a request / API call • Requires understanding of internals 34
  35. 35. More Considerations • Requires a load test tool license for the necessary number of users • Environment may need to be set on all agents • May require more resources on agents • Results should be cautiously interpreted 35
  36. 36. Example 1: Essbase Query • Multi-Dimensional Database • C API – Used by many applications and middleware – Winsock scripts • Quite difficult to parameterize and verify • External DLL was made for major functions 36
  37. 37. Winsock Script lrs_create_socket("socket0", "TCP", "LocalHost=0", "RemoteHost=ess001.hyperion.com:1423", LrsLastArg); lrs_send("socket0", "buf0", LrsLastArg); lrs_receive("socket0", "buf1", LrsLastArg); lrs_send("socket0", "buf2", LrsLastArg); lrs_receive("socket0", "buf3", LrsLastArg); lrs_save_searched_string("socket0", LRS_LAST_RECEIVED, "Handle1", "LB/BIN=x00x00vx00x04x00", "RB/BIN=x04x00x06x00x06", 1, 0, -1); lrs_send("socket0", "buf4", LrsLastArg); lrs_receive("socket0", "buf5", LrsLastArg); lrs_close_socket("socket0"); 37
  38. 38. Winsock Script send buf22 26165 "xffx00xf0a" "x00x00x00x00x01x00x00x00x01x00x03x00" "dx00bx00" "y'<Handle1>x00" "brx00x06x00fx00x1bex00x00rx00xd6aRN" "x1ax00x06x00x00x00x00x00x00x00x00x00b" "x00x00x00xe7x00x00x01x00x03x00x04x00" "x10x00xccx04x05x00x04x00x80xd0x05x00t" "x00x02x00x02x00bx00<x00x04" "FY04aWorkingtYearTotaltELEMENT-FtProduct-P" "x10<entity>tx00x02x00" … 38
  39. 39. Script Using External DLL lr_load_dll("c:templr_msas2k.dll"); pCTX = Init_Context(); hr = Connect(pCTX, "ess01", "user001","password"); … lr_start_transaction("Mdx_q1"); sprintf(report, "SELECT %s.children on columns, %s.children on rows FROM Shipment WHERE ([Measures].[Qty Shipped], %s, %s)", lr_eval_string("{day}"), lr_eval_string("{product}"), lr_eval_string("{customer}"), lr_eval_string("{shipper}")); hr = RunQuery(pCTX, report); lr_end_transaction("Mdx_q1",LR_AUTO); 39
  40. 40. Example 2: EDS • Essbase Deployment Services • Middleware, no GUI interface • Functional test scripts in Java • Solution - creation of LoadRunner scripts from the functional test scripts 40
  41. 41. EDS Java Script import lrapi.lr; … public int action() { lr.start_transaction("01_Create_API_instance"); ess = IEssbase.Home.create(IEssbase.JAPI_VERSION); lr.end_transaction("01_Create_API_instance", lr.AUTO); lr.start_transaction("02_SignOn"); IEssDomain dom = ess.signOn(s_userName, s_password, s_domainName, s_prefEesSvrName, s_orbType, s_port); lr.end_transaction("02_SignOn", lr.AUTO); … 41
  42. 42. Agenda • Load testing and its role in performance risk mitigation • Different approaches: load generation • Different approaches: environments • Load testing tools 42
  43. 43. Environments • Lab vs. Service (SaaS) vs. Cloud (IaaS) – For both the SUT and load testing tool • Test vs. Production • No best solution, depends on your goals / system 43
  44. 44. Scenarios • System validation for high load – Outside load (service or cloud), production system – Wider scope, lower repeatability • Performance optimization / troubleshooting – Isolated lab environment – Limited scope, high repeatability • Testing in Cloud – Lowering costs (in case of periodic tests) – Limited scope, low repeatability 44
  45. 45. Agenda • Load testing and its role in performance risk mitigation • Different approaches: load generation • Different approaches: environments • Load testing tools 45
  46. 46. Load Testing Tools • Differ drastically – Supported approaches / protocols – Scripting / extendibility – Supported environments – Scalability – Integration – Result analysis – Environment monitoring – Cost/Licensing – Available support and skills 46
  47. 47. Load Testing Tools • There is no best tool – it depends on your needs • Almost every tool will work for a simple web site • If using more sophisticated technologies, always check if the tool / approach supports it 47
  48. 48. Summary • Load testing is an important way of performance risk mitigation / part of the performance engineering process – Other ways don’t substitute load testing • There are many ways to do load testing. • There is no best approach or tool – it depends on your needs. 48
  49. 49. Any questions? 49 Alexander Podelko alex.podelko@oracle.com www.alexanderpodelko.com @apodelko
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×