This document discusses best practices for performance testing in Agile and DevOps environments. It recommends implementing early and continuous performance testing as part of CI/CD pipelines using tools like JMeter and WebPageTest. Additionally, it stresses the importance of system-level performance tests during targeted sprints and prior to production deployments. The use of application performance management tools to monitor tests and production is also highlighted to facilitate quick feedback loops and issue resolution.

1. Performance Test Strategy Best Practices
Early (Continuous) Vs System level Performance tests
Ramya R Moorthy

2. Agile versus DevOps Performance Testing
Building Continuous Performance Testing Framework
Early (Continuous ) vs System Performance Tests – Do’s & Dont’s
Case Study

3. Introductionto Agile & DevOps
 Strictly “Performance is everyone’s responsibility”
 Agile - Promotes continuous iterative software development methodology
 DevOps - A culture where Dev, QA and Ops collaborate with automation being the
crucial success factor (Deliver Fast with high Quality)
Shift from Traditional (reactive) Waterfall performance Testing to
 Agile (Be Proactive / Test Early on all iterative releases)
 DevOps (Test Continuously / Automate as much as possible)
Most Performance Testing tools now support DevOps (provides plug-ins for CI server)

4. Little Background
Requirement
Analysis
Requirement
Analysis
DesignDesign ImplementImplement TestTest
Deployment &
Maintenance
Deployment &
Maintenance
Traditional (Waterfall)
Performance testing
Agile (still Waterfall)
Performance Testing
Sprint 1Sprint 1 Sprint 2Sprint 2 ...Sprint 8...Sprint 8
Hardening
Sprint
Hardening
Sprint
Agile (Early)
Performance Testing –
Targeted Sprint
Sprint 1Sprint 1 …Sprint 4…Sprint 4 … Sprint 8… Sprint 8
Hardening
Sprint
Hardening
Sprint
Performance
Testing

5. Little Background
Sprint 1Sprint 1 Sprint 2Sprint 2 Sprint 3Sprint 3 Sprint 4Sprint 4
Hardening
Sprint
Hardening
Sprint
Scrum Team
Sprint 1
Perf Test
Sprint 1
Perf Test
System level
Perf Test
System level
Perf Test
Performance Team
Sprint 2
Perf Test
Sprint 2
Perf Test
Sprint 3
PerfTest
Sprint 3
PerfTest
Agile (Early)
Performance Testing –
Lazy Sprint

6. Little Background
Sprint 1Sprint 1 Sprint 2Sprint 2 Sprint 3Sprint 3 Sprint 4Sprint 4
Scrum Team 1
Sprint 1Sprint 1 Sprint 2Sprint 2 Sprint 3Sprint 3 Sprint 4Sprint 4
Scrum Team 2
Sprint 1
Integrated Build
Performance Testing
Sprint 2
Integrated Build
Performance Testing
Sprint 3
Integrated Build
Performance Testing
Sprint 4
Integrated Build
Performance Testing
Agile (Early)
Performance Testing –
Within Scrum
Note : Performance Analysts within Scrum Team + Scrum of Scrums
Need for CI Performance Tests realized

7. Areyou readyfor Agile/ DevOps Culture?
Not as easy as switching between Technologies
CULTURAL SWITCH
Requires SHIFT in PEOPLE + PROCESS + TECHNOLOGY
Waterfall
Agile
DevOps
• Number of PROD releases
• Faster Feedback Cycle
• Automation scope
• Collaboration Effort
It can take
years for the
complete
transition

8. Rememberthis Golden Rule
NO – ONE – SIZE – FITS – ALL
There is NO single standard performance testing
methodology that’s fits all Environments
Add value by
Early & Continuous & Automated
Performance analysis
YOU decide how to do that…
(that works best for you & your project)
CI / CD / CD
??

9. Continuous IntegrationLoad Testing Pipeline
Project Case Study : CI Load Testing pipeline (using Jenkins + JMeter + WebPageTest)

10. ModernPerformance TestingFramework
 Define Performance Test objectives / SLAs (into user stories)
Strictly include Performance in ‘DoD’ & Performance analyst as part of scrum team
 Define when / what to Test & its KPIs
 Continuous Integration (CI) Performance tests on nightly builds
 Sprint level : New Feature Performance tests / Regression Performance Tests
 Hardening Sprint : System level Performance tests
 Clearly Define Automated Test Result ( Pass / Fail ) Criteria
 Transaction Response Time (Average or 90th percentile value)
 Web page loading / rendering Time
 Acceptable Error rate (Transaction failures)
 Server throughput (Hits / sec or Transactions/sec)
 Server CPU & Memory utilization level

11. ModernPerformance TestingFramework
 Define what to test in various environments (which s/w version? & part of CI pipeline? )
Pre-Prod / QA / Staging / UAT
Choose the right performance testing tool
 Integration with CI server (like Jenkins/Bamboo)
 Integration with reporting tools (like Kibana/ Graphana / influxDB)
 Integration with APM Tools
 Handy & automated Performance Test Report Dashboards
 Quick & on-demand scaling of virtual users
 Quick & automated Load Generator Setup (using Docker container)
 APIs to Read / Share Test data from open source tools
Choose the right APM tool
To monitor test & production environments (full stack monitoring)
To perform Performance Engineering analysis (on Containers / Servers / Cloud, etc)
To bring shift+ right techniques to create early & quick feedback loops

12. You Decide When/ Whatto Performance Test ?
Continuously Evolving Performance Test strategy
[ APIs  Components  Microservices  Individual Use cases  System-wide ]
 Every Sprint (on Dev or QA Environment) :
 Quick CI performance tests to validate against baseline
 Baseline API/component/query performance for 1 user
 Validate (thru short tests) the feature for small load levels
 Targeted Sprint (on Staging / Perf-Test Environment) :
 Regression Testing for Integrated builds
 Regression Test suite continuously evolves
Hardening Sprint (on Perf-Test / Pre-Prod Environment) :
 System level Tests - Load , Stress, Spike , Soak, etc
 Use realistic workload model to validate the performance, scalability & capacity
to handle real world usage conditions.

13. Once Luxury...But Now Mandatoryin DevOps
 Early & Continuous Integrated Performance Tests (automate as much as possible)
 Continuous Performance Monitoring (Test + Prod environments)
Shift + Left (Supporting Ecosystem : Service Virtualization tools & Perf Test tools)
 APIs
 Components
 Microservices
 Custom ML / AI implementations : Trend analysis / detect build-wise performance degradations
 Shift + Right (Supporting Ecosystem : APM tool , RUM tool, etc)
 RUM metrics to build/ validate workload model
 Quick Feedback Cycles reduce problem resolution time from days to hours to minutes.

14. Choose right Environment (for PerformanceTesting)
You Strategize wisely as per your needs!!
CI Pipeline till Sprint #5
CI Pipeline from Sprint #6

15. Performance EngineeringResponsibility
Load testing in CI pipeline will become incomplete
If No Continuous monitoring & automated diagnosis by APM tools

16. Performance EngineeringResponsibility
Reporting increase in response time / server CPU Utilization
Is this enough in nightly CI builds?

17. Do’s & Dont’s inEarly(continuous) PerformanceTests
 Run early performance tests (API / Unit level) as part of CI pipeline
Use Stubs / Service Virtualization / Custom implementations
Measure performance for 1 user load & small user loads (don’t over test / over measure)
 Run performance tests on small scale environments like Dev , QA or Staging
Do not include long duration tests in CI pipeline
Monitor key metrics to define build pass / fail criteria & toggle detailed metrics on demand
 Use APM tool to facilitate quick feedback loops with PE recommendations
Realistic Test data volumes / think time settings / transactional throughput not important

18. Do’s & Dont’s inSystem LevelPerformanceTests
Ensure long duration regression tests are carried out on targeted sprints
Continuously improvise Test scripts & Workload model to meet realistic usage patterns
Run system level tests on Prod similar environments with realistic think time & test data
 Use APM tool to provide detailed analysis to quickly fix the issues.
 Add APM monitoring as part of CI pipeline to promote the build
 Improvise system level test suite based on feedback loops from production environment
APM tool & RUM tool metrics.
 Implement custom made cool performance dashboards (with heat maps using data
visualization plug-ins) & follow up the results to quantify improvements

19. DevOpsPerformance Testing Case Study Highlights
(ECommerce Application)
 Performance Test SLAs ?
 What Tests Executed ?
 Early Performance tests coverage ?
 System level Performance tests coverage ?
 Which Environment Used ?
 Automated Test Results Dashboards ?
 Toolset (Testing + Engineering) used ?

20. CI/CD PerformanceToolset / Environments
Jenkins ProdQA Staging Pre-ProdDev
Sonar Code
Analysis
&
JUnit Tests
Selenium
Tests
&
Web Page
Test
Selenium Tests
&
Low Load Test
(APIs)
Smoke Test
API Load Test
&
Low Load Test
System Load Test
Stress Test
Spike Test
Endurance Test
CI Pipeline Test
APM
Monitoring
CI Pipeline Test
&
Independent
Load Tests
CI Pipeline Test
&
Independent
System Tests
JMeter + Selenium + Taurus + WebPageTest + Dynatrace

21. Code Analysis
Jenkins Plug-ins :
(Checkstyle / PMD / FindBugs / SonarQube Scanner )
Custom
Rules
Set

22. Web Performance Analysis
Jenkins Plug-ins : Sitespeed plugin (Sitespeed.io)
(using WebPageTest APIs through InfluxDB & Grafana)

23. Performance EngineeringAnalysis
APM Tool (Dynatrace - Performance Signature Jenkins Plug-in & One Agent Monitoring)

24. Summary
 Understand the big picture "The WHOLE system"
 Cultivate early performance analysis culture
 Add early & continuous feedback loops
 Do not compromise system level performance tests (due to early CI tests)
 Continuous Performance Testing approach one & only if combined with performance
diagnosis tools (Performance Engineering) can aid in fast delivery in DevOps environment.