Neotys organized its first Performance Advisory Council in Scotland, the 14th & 15th of November. With 15 Load Testing experts from several countries (UK, France, New-Zeland, Germany, USA, Australia, India…) we explored several theme around Load Testing such as DevOps, Shift Right, AI etc. By discussing around their experience, the methods they used, their data analysis and their interpretation, we created a lot of high-value added content that you can use to discover what will be the future of Load Testing. You want to know more about this event ? https://www.neotys.com/performance-advisory-council

1. Performance Testing in a DevOps world: Don't spin in the middle
Stijn Schepers

2. Magritte
A month ago, my wife asked me to join her to visit an exposition of Magritte in
Brussels. Magritte is a famous surrealistic Belgian painter. When we entered the
museum, a black-white movie of Broodaerts – who was a friend of Magritte - was
playing. Broodaerts was writing with an exceptional technique letters on a piece of
white paper. Suddenly it started to rain; his writing technique did not change. The
technique was still perfect but the result of this perfect technique was not there
anymore. The rain made the text blurry. When I saw this movie I was thinking that as a
performance test engineer I have felled like this for a long time. Not too long ago,
many projects and programmes of work were delivered based on a waterfall delivery
model (SDLC). Typically these projects took months or sometimes years to deliver and
were often delivered with significant delays, rarely on-time. Performance testing
would typically happen at the end of the lifecycle and was often squished to a test
window just days before go-live. There was no room for optimization or improving the
system from a technical point of view and the programme would go-live with
technical dept.

3. Luckily most projects are now delivered in an agile way
based on DevOps principles.
We try to deliver a project in a continuous and automated
fashion; development, testing and operations are working
closely together with a focus to increase the speed of the
train or de cadence of the releases. Projects are not being
delivered as a big bang but small features are being
released when they are ready to be shipped.
As the release cycles are getting shorter and shorter, we do
need to change the way we do testing. The time we have
to script “load scenarios” and to run tests is less. The way
we test in a DevOps world is therefore different.

4. L
E
F
T
RI
GH
T
Testing in a DevOps world: Don’t spin in the middle –
shift left and shift right.
Limiting performance testing to E2E performance
testing is a NO NO. E2E scripts often fail and are very
sensitive to changes. The risk with E2E testing is that
all your time will be spend doing scripting without
bringing any value to the project.

5. TEST IN PRODUCTION Shift right
RIGHTRIGHT
In DevOps there is always a lot of focus on SHIFT LEFT , testing closer to the build and the design phase of the
features. It’s my believe that SHIFT RIGHT is equally IMPORTANT for performance engineering. We should TEST
in production.
Production is the only environment that is “production like” and the real users are testing the system. We do
not have to pay these users, quite often they pay to use the system (online shops, trading platforms, etc). So
why should we not use the information that they leave behind? It’s free and it is these traces provide us with
insights of the production system. Everything a user does can be found in log files and can be analysed with BI
tools. Or even better, by having Application Performance Monitoring (APM) tools (AppD, DynaTrace, New Relic)
in your production environment. These tools will monitor the performance of your business transactions
through the system components and will tell you exactly in which line of code the majority of the time is
spend. So after a release you get immediately feedback about the impact the release had on the performance
and the stability. A performance engineer should have FULL access to the APM tools of production because
this info provides the engineer with crucial information about how the application performance and where the
focus of testing should be on.

6. These APM tools have changed my life as a test engineer. The tools
should be installed in Test and Development. It provides invaluable
information so instead of raising issues we can now provide solutions to
these issues. Think about it….
The mindset and joy you get out of providing solutions is so much higher
than reporting defects (that may often be false positives ). The value of
testing is so much higher when load test tools can be integrated with
APM tools.

7. LEFT
We need to shift left as well. The E2E scripts are difficult and sensitive to changes (releases). Therefore, I would recommend
to build up an automated regression framework based on Soap or Rest calls (API). Luckily most modern enterprise apps do
use APIs. Scripting on Interface level is less difficult and the scripts are way more robust and are not as sensitive to front-
and back-end changes. The ratio of load (service calls) can be retrieved from APM tools in production OR by analyzing log
files of the enterprise bus. BI Tools like Tableau are very valuable for doing this kind of smart analysis.

8. When your delivery framework is Scaled Agile , a Product Increment Day (PI-Day)
is a perfect opportunity to do a risk assessment on the features that are being
added to the backlog for the next release. It is extremely important that the focus
of performance engineering is on feautures that impose the highest risk to the
business. Often there is a very limited pool of resources with deep knowledge of
performance engineering so it is key to use the time as valuable as possible. Put
your eggs in the right basket! Important is to have a NFR defined in the definition
of DONE. So scrum teams have a clear focus on performance. When scrum teams
are put together based on Business Value Streams it helps to define meaningful
easy to understand performance requirements (DOD).

9. DevOps is about CONTINUOUS: continuous integration, continuous delivery, continuous
feedback, continuous testing and continuous performance testing. How I see continuous is
not that we will be doing performance test execution every single second of the day in a
fully automated way. What continuous means to me is that we have a team of performance
engineers that will be involved with testing the features of a release train continuously
without an end-date (<> big bang – V-model). The team will get a better understanding of
the “systems under test” so the engineers can improve the way they do testing
continuously. During these release cycles, the “technical debt” of the testing frameworks
and test approaches can be reduced. Continuous means do better testing each release
cycle and manage IP in a more efficient way.

10. APM
Synthetic Monitoring
RISK Assesments
Automated SOAP/REST testing
CI - Perf Test
E2E Performance Validation
(white box – APM)
***A U T O M A T E D ***
Feedback loops
Summary :
Shift Right: APM + Synthetic monitoring
Shift Left: Risk Assessments + Automated API (REST – SOAP) testing + CI with Perf Testing
For CI testing: be extremely careful with false positives!! You need a very stable “system under test” to be
able to do this efficiently.
(Code as infrastructure)

11. With DevOps automation is key to accelerate the release cycles and increase the cadence. Automation for performance testing is
possible but the analysis of results requires the analytical eyes of an experienced test engineer. A test engineer in a DevOps world
must have Business Intelligence (BI) tools in their toolkit. With more load test tools moving to the cloud it is important to emphasis
the importance of raw data. Raw data means that you do your result analysis based on every single measurement AND NOT based
on aggregated results (Averages). Aggregation hides patterns and hides bottlenecks. Unfortunately some “load test tools in the
cloud” only work with aggregation and do lose some of their value.

12. Artificial Intelligence and machine learning will further change the way
we do performance testing. The time required to do test scripting
needs to further decrease. Why do we still require to do scripting at all if
we have APM tools monitoring all traffic in production. Can these APM
tools not be used for automated script generation, workload modeling
and data modelling? Also we want these APM tools to tell us when
issues will occur, not when they have occurred. With so much data
provided to APM tools, these can in theory be used (machine learning)
to automatically optimize applications (example change the heap size
or GC algorithm of a java app) based on algorithms and lots of data.
Why do APM tools not help us predict the cost of our cloud hosting and
provide us with measurable values to decrease these hosting bills?
For me it feels like performance testing in a DevOps world creates more value. As a performance test
engineer, there are less rainy days when the rain ruins my beautiful paintings.