Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Load testing with Visual Studio and Azure - Andrew Siemer

4,779 views

Published on

In this presentation we will look at what web performance testing is and the various types of testing that can be performed. We will then dig into Visual Studio 2013 Ultimate to see that the Visual Studio platform is now a real contender in performance testing automation. And we will see how the Visual Studio integration with Visual Studio Online and Azure can take your web performance tests and spin up impressive load tests in a truly useful way.

Published in: Software
  • Be the first to comment

Load testing with Visual Studio and Azure - Andrew Siemer

  1. 1. Performance testing with Visual Studio & Azure How to test your application using the tools you already know! Andrew Siemer | Clear Measure andrew@clear-measure.com @asiemer
  2. 2. Andrew Siemer http://about.me/andrewsiemer ASP Insider MS v-TSP (Azure) Azure Advisor Program Father of 6. Jack of all trades, master of some.
  3. 3.  We are hiring!!!
  4. 4. Introduction
  5. 5. Agenda • What is performance testing? • How can Visual Studio help? • How can Azure make load testing easy?
  6. 6. What is performance testing?
  7. 7. What is it? • A type of testing intended to determine • Responsiveness • Throughput • Reliability • Scalability …under a given workload
  8. 8. Why would I care? • Are you ready to go to production? • Can you handle the expected load? • Do you have edge case bugs? • How much traffic causes your app to tip over? • Do you need to optimize something? • Have new features caused the app to support less traffic? • Do you have enough hardware to support known load? • What causes your application to fail spectacularly?
  9. 9. Three types of performance tests • Performance testing • Load testing • Stress testing
  10. 10. Performance testing • Speed • Scalability • Stability • Achieves expected performance in general use case • Can be run often • Cheap to run with low volume
  11. 11. Load testing • Expected performance remains steady under production load • Assumes everything is normal • Does the network handle this load • Can the database handle it • Is the application still meeting SLA and usable
  12. 12. Stress testing • Pushing the application beyond expected limits • Identifies ceilings in capacity • Tests low memory, disk space limitations, or dead server • Helps see how and when an application will fail • What happens when the network gets clogged
  13. 13. Many other types of tests • Capacity (system capacity meets business volume) • Component (component meeting expectations) • Endurance (is performance consistent over time) • Investigation (performance trending over time) • Smoke (build ready for perf testing) • Spike (temporarily exceeds expected load) • Unit (segment of code reasonably efficient) • Validation (faster or slower)
  14. 14. Baselines • A baseline tells you where the app is now • Allows you to see change over time • Target to demonstrates improvements • Baselines can be created for • System • Component • Application
  15. 15. Benchmarking • The comparison of a current test with a baseline • Looking for changes in the results • Or a comparison against industry standards • Are my commerce pages as fast as the industry expects
  16. 16. When to shift from testing to tuning? • Once testing has found unacceptable results • When results are acceptable, but servers are running extra hot • When an SLA is breached that we know we need to fix • Especially when it impacts a large test surface
  17. 17. Test planning
  18. 18. Before we test • Define acceptance criteria • Write down key scenarios • Create workload model • Target load levels • Determine metrics to capture • Design tests
  19. 19. Acceptance criteria • Write down your objectives • Response time – page load speed • Throughput - number of concurrent users • Resource utilization – processor, memory, disk I/O, network I/O • Maximum user load • Business metrics – number of orders to handle
  20. 20. User scenarios • “Happy paths” through your application • Touches many components in your system • Commonly used paths through the application • Highest business value paths through the application
  21. 21. User scenarios - examples • Log in • Browse catalog • Add to cart • Check out • Register • Search • Faceted navigation
  22. 22. User scenarios – specific example • For a given scenario – determine activities in the test • The scenario “add to cart” may include these activities • View home page • View dirt bike category page • Choose grips • Add a grip to the cart • Go to the cart • Validate that the item is in the cart
  23. 23. User scenarios – most common/intensive • Be sure to include highest use scenarios • And include most resource intensive scenarios • Look at your existing applications log files for top use case
  24. 24. Define the workload • Know and attempt to simulate existing traffic patterns • Understand user delay, or “think” time • A load test should not be a denial of service attack! • Plan scenarios around average session times • Not too short or too long • Not every scenario can be a new user, returning user, or either • Plan around the reality of your application
  25. 25. Target load levels • Load levels are applied to “workload” • Understand business volume as it relates to your objectives • Common load, vs. a big marketing push, or black Friday • Key scenarios • Distribution of work • Average session times
  26. 26. Define what metrics to capture - business • Too many metrics can make the results hard to read • Ask questions related to performance that have specific answers • How many orders are place per minute • What is the response time of the cart page • Identify what metrics to capture based on your questions • Looks for lower level metrics that help answer your questions • Reevaluate this often – applications change – so should the metrics
  27. 27. Define what metrics to capture - application • Network – hardware, switches, routers, gateways, load balancers • System – disk, processor, memory, network • Platform – the host of your app: IIS, worker role, web role, VM • Application – perf counters, instrumentation, file locks, db locks, queue
  28. 28. Design tests • Using your thought so far design specific tests to meet your needs • Don’t change the tests because it is hard to write it as designed • Test expected data as well as unexpected data (form validation, bad credit card, etc.) • Be sure to include think time • Best tests data is collected from production data (where applicable) • Think about spiders, batch processes, while real users are browsing • Don’t over simplify your tests!
  29. 29. How can Visual Studio help?
  30. 30. Visual Studio 2013 Ultimate • VS feature matrix: http://goo.gl/3VtY01 • Ultimate gets you web load and performance testing • Test Pro doesn’t get you this functionality!
  31. 31. Web performance tests The beginnings of our load testing!
  32. 32. The test project
  33. 33. Recording a test
  34. 34. Recording a test – uh oh! • The first time you try to record, IE likely won’t work • Basically, you need to • Disable Enhanced Protection Mode • Enable Web Test Recorder • Go here for full steps to fix this: http://goo.gl/LcFNSJ
  35. 35. Recording a test – uh oh!
  36. 36. Recording a test
  37. 37. Recording a test
  38. 38. Recording a test
  39. 39. Verbose recording…likely too much!
  40. 40. Let’s clean up the recording
  41. 41. Pruned tests
  42. 42. Recorded tests cleaned up
  43. 43. Run the tests locally
  44. 44. Run the tests locally
  45. 45. Modifying test run settings
  46. 46. Making dynamic tests Way more useful!
  47. 47. Once you start seeing repetition
  48. 48. Consider investigating patterns in the tests • Look for patterns in the requests • Create a data source with variable data • Sku’s • Makes • Models • Year • Dynamically create a wide set of tests
  49. 49. Adding a data source • Once tests need to cover more… • Or you need to create representative load… • Types of data source • Database – “select * from tablename” • CSV • XML
  50. 50. Create a shippable datasource
  51. 51. Add the data source to your project
  52. 52. Point at the file then magic
  53. 53. Choose the table to include
  54. 54. And then we have a data source
  55. 55. Update recording to use data source
  56. 56. Update recording to use data source
  57. 57. If the test only runs once…
  58. 58. View the dynamic results
  59. 59. Conditional & loop steps • Context parameter exists (test context) • Cookie exists • Cookie value comparisons • Last request outcome • Pass or fail • Last response code • http response codes • Number comparison • String comparison • Probability rules • Return true some times • Loops
  60. 60. Plug ins • Web test plug ins • Web test request plugins • Should be in a separate assembly • Allows you to run code pre or post a test/request execution • Microsoft.VisualStudio.TestTools.WebTesting • WebTestPlugin • WebTestRequestPlugin
  61. 61. Load tests
  62. 62. Web tests to the max!!! • Use the web test scenarios you create • But put the workload, load, metrics, browser mix, and other points around them
  63. 63. Load test wizard • Running the load test wizard makes life easy!
  64. 64. When matching transaction percentages
  65. 65. When matching percentage of users per test
  66. 66. When matching pace of user per test
  67. 67. When matching pace of user per test
  68. 68. Choose to tests to include
  69. 69. Set distribution for each test
  70. 70. Define network mix
  71. 71. Define browser mix
  72. 72. Capture computer metrics
  73. 73. Configure the run frequency
  74. 74. Ready to run!
  75. 75. Errors?
  76. 76. But wait there’s more!
  77. 77. How can Azure let you go big?
  78. 78. Your laptop can’t tip over production! • Local tests are great to capture big pain points • Or to capture simple content missing errors • Or to run as a smoke test
  79. 79. Test farm in Azure is magic! • With Visual Studio Online • And Azure • We can go big!
  80. 80. Open local test settings
  81. 81. Choose to run in VSO
  82. 82. Connect to your VSO account
  83. 83. Choose your team project
  84. 84. And run the test!
  85. 85. Ok, that rocks, HOW MUCH? • Based on “Virtual Users Minutes” – VUM • Azure calculator for load testing: • http://goo.gl/XBrkY1 • $.0004/VUM for 20,001-2M VUM • $.0002/VUM for 2,000,001-10M VUM • $.0001/VUM for usage above 10M VUM/MO
  86. 86. Questions? Andrew Siemer - Clear Measure andrew@clear-measure.com (512) 387-1976 @asiemer

×