Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Load Testing using Continuous Integration tools


Published on

From 2015 Neotys North American User Conference
Speaker: Richard Pitts
Customer case study of CommerceHub's use of NeoLoad via CI Tools.

Published in: Software
  • Be the first to comment

Load Testing using Continuous Integration tools

  1. 1. LOAD TESTING using continuous integration tools Richard Pitts Sr. Load Test Engineer | CommerceHub
  2. 2. Agenda  Our experience adding NeoLoad to our Continuous Integration process. 1. Benefits of using CI to run NeoLoad 2. Why we run NeoLoad using CI 3. How NeoLoad + CI has helped us 4. Common pitfalls to avoid 5. Getting started + best practices
  3. 3. About CommerceHub  Hosted services for e-commerce & retail.  [B2B] Connection between 200+ retailers and 8,900 brands/distributors.  Translate & route drop shipment orders amongst business partners.  Goal: Facilitate e-commerce sales growth for our customers  Processed $10bil GrossMerchValue for our customers in 2014 [33.7mil orders]  Why performance matters to us:  “retail busy season”: Black Friday  Christmas  38k+ users  8.25 mil logins/yr  2 mil hits per month on most visited web page
  4. 4. Benefits of using CI to run NeoLoad Same benefit as any CI Job  Easy to execute  Easy to schedule  Easy to monitor  Build history  Results in Workspace
  5. 5. Benefits of using NeoLoad Plugin  Retrieve Neoload results back into to Jenkins  Display Trends: Avg Response Time & Error Rate  Maintain build history & preserve Neoload reports for each build.
  6. 6. Benefits of using NeoLoad Plugin Collaborate. Easy to share. Link to the build.
  7. 7. Why you should run NeoLoad using CI  If load testing occurs less frequently than desired  New development methodologies  more frequent releases  Scheduled automation avoids waiting on humans, keeps a faster pace.  Our Story…  before NeoLoad  with NeoLoad  with NeoLoad plus CI
  8. 8. Load Testing before NeoLoad • Close to the release date • Late testing  Late Discoveries  Delayed Releases. feature complete build Monthly releases Performance test @ end of dev cycle
  9. 9. Load Testing before NeoLoad  An automated tool could help us load test.  Hopefully one that would inspire us to load test earlier + more often.  Started to Trial + Assess several load testing tools
  10. 10. NeoLoad pilot  Began recording & designing our Neoload scripts
  11. 11. A better Load Test environment  Properly designed for load testing @ high volume [150+ users]  Used CI to automate deployment & configuration of this env  On-premise Load Generators  Web cluster behind Load Balancer
  12. 12. NeoLoad pilot: real progress  Started generating performance test results
  13. 13. Success  implement an automated load testing tool  library of scripts  now easier to include load testing for releases  load testing more things, more often  stockpile load test results for future benchmark comparisons
  14. 14. NeoLoad + Waterfall: How to? So, how did we regularly include NeoLoad tests as part of our waterfall releases?
  15. 15. NeoLoad + Waterfall: Option 1 Three options to include Neoload testing for a release: 1) author new NeoLoad scripts designed to target specific concerns.  Record/Playback  Configure for multi-user concurrency & unique data  Run scripts  Share results
  16. 16. NeoLoad + Waterfall: Option 2 Three options to include Neoload testing for a release: 2) run existing NeoLoad scripts + compare w/ previous release results  Try/validate the scripts  Update/fix outdated scripts  Run scripts  Share results
  17. 17. NeoLoad + Waterfall: Option 3 Three options to include Neoload testing for a release: 3) run NeoLoad scripts on project vs mainline branches, comparing results  Run scripts on mainline branch to obtain baseline results  Run scripts on project branch  Compare project results to baseline results  Is performance of the project branch better/no worse/worse than mainline
  18. 18. Results: Still a Waterfall  Performance testing still occurred @ end of dev cycles  Limited time to re-work performance issues before release dates.  Often, the tests were short samples due to time constraints.  The most recent set of results for comparison was often many weeks old.  Centralized around performance engineer (bottleneck risk)  Sometimes it was “testing for the sake of testing”
  19. 19. Meanwhile: Along comes Agile  Continuous Integration of code  Automated CI builds  Continuous Delivery to test environments  Automated regressions using CI builds  Cross functional team members  “Everyone is responsible for quality” approach  Result:  Sped up development + shortened our dev lifecycle! Release more frequently
  20. 20. Load Testing + Agile Teams  Team mindset: “Performance is everybody's job”  But, most team members had little load testing experience.  Result:  Still had all of the same problems getting Load Tests done.  Completed “in the margins” by perf engineer + agile team member.  Relied upon Ops and/or performance engineer (bottleneck risk)  Load testing wasn’t keeping pace with our Agile teams.  Needed a way to bake Load Testing into our Agile process.
  21. 21. Load Testing + Agile Teams  How could we include LT more often, under Agile process?  Manual performance testing? [+leverage firebug, Yslow, Fiddler, etc.]  Collect performance metrics while we run our automated regression?   Schedule NeoLoad to run using Cron jobs?  Schedule NeoLoad to run using NeoLoad internal scheduler?  Schedule Neoload to run using CI? Hmmmm… we do have Jenkins.
  22. 22. Jenkins NeoLoad plugin v4.2 [Nov 2013]  Retrieve Neoload results back into to Jenkins  Display Trends: Avg Response Time & Error Rate  Maintain build history & preserve Neoload reports for each build.
  23. 23. Gains from NeoLoad + CI Daily performance “pulse”
  24. 24. Gains from NeoLoad + CI JUnit test results based on SLA profiles
  25. 25. Gains from NeoLoad + CI Collaborate. Easy to share. Link to the build. Increased Collaboration.
  26. 26. NeoLoad Tests: Controlled by CI Scheduled nightly.
  27. 27. Lessons learned Pitfalls to Avoid:  Using only “happy path” scripts  Comparing results over long gaps time  risky  Comparing multiple short samples  inaccurate  Swap sharing the Load Testing environment  bottlenecks  Requiring involvement from Ops or perf. engineer  bottlenecks
  28. 28. Your turn!  Run NeoLoad using Continuous Integration
  29. 29. Step1: get a CI Server, setup for NeoLoad We used an existing remote Jenkins server. Use your existing remote CI server or install Jenkins along side NeoLoad.
  30. 30. Step2: add CI step to run NeoLoadCmd Neotys’ Jenkins Integration Guide explains how to configure –options
  31. 31. Step3: Install NeoLoad plugin for Jenkins
  32. 32. Step4: retrieve Neoload results Jenkins Integration Guide explains how to incorporate NeoLoad results
  33. 33. Where to go from here? Best practices & Pro Tips:  Incorporate performance testing into every DevOps team  Intelligent testing: only run load scripts that test the code that was changed  Reduce the amount of time & resources needed for load testing  For “hardening sprints”- setup a Stress test scenario. Run on-demand using CI  Make use of email alerts and chat notifications in CI
  34. 34. Best Practices: Setup SLA profiles  Setup SLA profiles so pass/fail criteria are bubbled up to CI results
  35. 35. Best Practices: Increase collaboration
  36. 36. Best Practices: Smoke test  Smoke test first (low volume), then run full Load Test
  37. 37. Richard Pitts We are hiring: Share with me: How are you using CI to drive your NeoLoad process? Let me know. Learn more: