Performance testing with VSTs on- and off-premises


Published on

Many of us have been there before... You deploy your brand new web site to production, you know, the one that's going to make your company rich and you famous, and just as you're about to hit a thousand users, the site comes crashing down! If you haven?t experienced this, chances are you know someone who has.If you're building web apps and services for your organization you know how important it is to ensure those services are always up and running. While many of us go through great lengths to ensure we are testing the functionality of our systems we do not always take performance or system load into account.Visual Studio Team Services (VSTS) provides multiple options and avenues for enabling performance and load testing for your systems. For example, did you know that VSTS also supports JMeter tests? With VSTS you can even run performance tests using on-premises servers in your own data center. This talk will give you an overview of the performance and load testing capabilities of VSTS. If you haven't looked at the load testing features of VSTS lately, you should definitely check out this talk!

Published in: Software
  • Be the first to comment

  • Be the first to like this

Performance testing with VSTs on- and off-premises

  1. 1. Performance Testing with VSTS On- and Off-premises Jeff Bramwell VP – Enterprise Architecture MVP – Visual Studio & Development Technologies | @jbramwell
  2. 2. Logistics
  3. 3. Please Be Courteous! Please be courteous to your fellow attendees and set your phone to vibrate or silent mode
  4. 4. Stuff to Cover ( i.e. Why am I here? ) • VSTS in a nutshell • Load test vs. Stress test – are they different? • Load test options in VSTS • Running load tests on-premises • Summary – go forth and break stuff! • And, along the way, lots of demos!
  5. 5. VSTS in a Nutshell Visual Studio Team Services (VSTS) is… • Microsoft’s web-based ALM offering (i.e. TFS in the cloud) • Agile planning tools (work items, kanban boards, etc.) • Source code repository (Git or TFVC) • Package management (NuGet & NPM) • CI/CD through automated builds and deployments • Test automation • Dashboards and widgets • Etc…
  6. 6. Load & Stress Tests • The terms Load Test and Stress Test are commonly interchanged • There is a difference: • Stress testing is intended to push a system to its breaking point • Load testing is used to determine how a system behaves under a specific load • Load tests can generally be configured as a stress tests • Load tests are commonly used to determine if a system can meet an SLA
  7. 7. Why Load/Stress Test? To determine: • How many users a system can support • If you can meet your SLA • The appropriate amount of (virtual) hardware resources needed for a system • How a system performs under normal usage as well as peek usage • Bottlenecks • Other performance issues such as memory leaks, race conditions, etc.
  8. 8. Load Test Options in VSTS • URL-based Test • HTTP Archive-based Test • Visual Studio Test • JMeter Test
  9. 9. On-Premises vs. Cloud Test Type On-Premises Cloud URL-Based Test Yes Yes HTTP Archive-based Test Yes Yes Visual Studio Test Yes Yes JMeter Test No* Yes *Supported by on-premises build agents but not yet supported by on-prem test agents
  10. 10. Deciding: On-Premises vs. Cloud Considerations: • Number of cores required to support testing needs • Geographic area(s) to be covered • Can cloud resources be utilized? • Sensitivity of data and/or regulatory concerns • Existing on-premises infrastructure • Frequency of test runs • Ramp-up requirements
  11. 11. Let’s Dig In!
  12. 12. URL-Based Tests • Easily and quickly setup load tests for a web site using VSTS • Requires Visual Studio Enterprise subscription (monthly, annual or MSDN) • As easy as: • Entering one or more URLs • Configuring a handful of run settings • Running the test
  13. 13. Demo URL-Based Tests -- In the Cloud --
  14. 14. On-Premises Test Agents • Can be installed and configured quickly via a single PowerShell command • Setup is fast • No need to open up non-standard (i.e. non-SSL/443) ports in firewall • Can configure as many on-premises test agents as needed (one per machine) • Test agents auto update (much like build agents) • PowerShell script available here:
  15. 15. Demo Test Agent PowerShell Script
  16. 16. Demo URL-Based Tests, cont’d -- On-Premises --
  17. 17. HTTP Archive-based Test • Similar to URL-based tests • Currently in preview (available to everyone except Stakeholders) • Can easily export .HAR files from popular browsers (e.g. via “F12 Tools”) • Fiddler can also export .HAR files • Includes URLs, headers, etc. • Allows you to create some basic web test scenarios very quickly
  18. 18. Demo HAR-Based Tests
  19. 19. Visual Studio Test • Can be based on: • UI recording • Unit Tests (MSTest only; NUnit not supported) • Not limited to just web scenarios • Much more configurable with lots more options • Highly customizable via code
  20. 20. Demo Visual Studio Tests
  21. 21. JMeter Tests • Can build on existing JMeter knowledge • Can make use of existing test suites • Lots of community expertise around JMeter • Very easy to integrate!
  22. 22. Demo JMeter Tests
  23. 23. Summary • Load/Stress testing: It’s not only good, it’s necessary • Lots of options! • Can be executed on-premises or in the cloud • Can setup load tests very quickly!
  24. 24. Additional Information • My Blog  - • Visual Studio Team Services - • VSTS Load Tests - us/docs/test/performance-testing/getting-started/get-started-simple-cloud- load-test • Test Agent Install Script: agents-on-your-infrastructure/
  25. 25. Burning Questions
  26. 26. Complete An Evaluation Form & Win! Your input is important! You can access Evaluation Forms at: Fill them out! You can win additional prizes! Like a $50 Best Buy Gift Card!! Winner drawn – Midnight, Sun Jul 23rd!