The document discusses the importance of continuous performance testing in software development, highlighting various benchmarks used to assess system performance, including nanobenchmarks, microbenchmarks, component benchmarks, and system performance tests. It emphasizes that process maturity in performance testing leads to increased confidence and maintenance costs while outlining strategies for effective monitoring and regression testing. Additionally, it advocates for using production-grade tools and methodologies to ensure accurate performance assessments and rapid feedback cycles.

1. Continuous
Performance Testing
Mark Price / @epickrram
Performance Engineer
Improbable.io

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
cd-performance-testing
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. The ideal
System performance testing as a
first-class citizen of the continuous
delivery pipeline

5. Process

6. Process maturity
A scientific and rigorous survey

7. Process maturity
A scientific and rigorous survey

8. Process maturity
“As part of QA, the whole team logs on
to the system to make sure it scales”

9. Process maturity
“We have some hand-rolled
benchmarks that prove our code is fast”

10. Process maturity
“We use a well-known testing
framework for our benchmarks”

11. Process maturity
“Our benchmarks are run
as part of CI”

12. Process maturity
“Trend visualisations of system
performance are available”

13. Process maturity
“There is a release gate on
performance regression”

14. Increasing process maturity
Implies:
Higher maintenance cost
Greater confidence

15. Scopes

16. Performance test scopes
● Nanobenchmarks
● Microbenchmarks
● Component Benchmarks
● System performance tests

17. Nanobenchmarks
● Determine the cost of something in the underlying
platform or runtime
● How long does it take to retrieve System.nanoTime()?
● What is the overhead of retrieving AtomicLong vs long?
● Invocation times on the order of 10s of nanoseconds

18. Nanobenchmarks
● Susceptible to jitter in the runtime/OS
● Unlikely to need to regression test these...
● Unless called very frequently from your code

19. Message callback
@Benchmark
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
public void singleCallback(final Blackhole blackhole)
{
callback.accept(blackhole);
}
@Benchmark
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
public void singleElementIterationCallback(final Blackhole blackhole)
{
for (Consumer<Blackhole> objectConsumer : callbackList)
{
objectConsumer.accept(blackhole);
}
}

20. Message callback

21. Microbenchmarks
● Test small, critical pieces of infrastructure or logic
● E.g message parsing, calculation logic
● These should be regression tests
● We own the code, so assume that we’re going to break it
● Same principle as unit & acceptance tests

22. Microbenchmarks
● Invaluable for use in optimising your code (if it is a
bottleneck)
● Still susceptible to jitter in the runtime
● Execution times in the order of 100s of nanos/single-digit
micros
● Beware bloat

23. Risk analysis - long vs double
BigDecimal
long
double

24. Component benchmarks
● ‘Service’ or ‘component’ level benchmarks
● Whatever unit of value makes sense in the codebase
● Wire together a number of components on the critical path
● We can start to observe the behaviour of the JIT compiler
(i.e. inlining)

25. Component benchmarks
● Execution times in the 10s - 100s of microseconds
● Useful for reasoning about maximum system performance
● Runtime jitter less of an issue, as things like GC/de-opts
might start to enter the picture
● Candidate for regression testing

26. Matching Engine - no-ops are fast!

27. System performance tests
● Last line of defence against regressions
● Will catch host OS configuration changes
● Costly, requires hardware that mirrors production
● Useful for experimentation
● System recovery after failure
● Tools developed for monitoring here should make it to
production

28. System performance tests
● Potentially the longest cycle-time
● Can provide an overview of infrastructure costs (e.g
network latency)
● Red-line tests (at what point will the system fail
catastrophically)
● Understand of interaction with host OS more important
● Regressions should be visible

29. Page fault stalls

30. Performance testing trade-offs
Nanobenchmarks
Microbenchmarks
Component Benchmarks
System Tests
● Slower
feedback
● Hardware
cost
● Maintenance
cost
● KPI/SLA
indicator
● Realism
● Faster
feedback
● System jitter
magnified
● Fewer moving
parts
● Stability

31. Measurement

32. System jitter is a thing

33. Reducing runtime jitter
Histogram of invocation times (via JMH)
Run-to-run variation
Large error values around average

34. Reducing runtime jitter

35. Measurement apparatus
Use a proven test-harness
If you can’t:
Understand coordinated omission
Measure out-of-band
Look for load-generator back-pressure

36. Production-grade tooling
Monitoring and tooling used in your
performance environment should be
productionised

37. Containers and the cloud
Measure the baseline of system jitter
Network throughput & latency: understand what is an artifact
of our system and what is the infrastructure
End-to-end testing is more important here since there are
many more factors at play adding to latency long-tail

38. Reporting

39. Charting
“Let’s chart our benchmark results so
we’ll see if there are regressions”

40. Charting

41. Charting

42. Charting

43. Charting
Make a computer do the analysis
We automated manual testing, we should automate
regression analysis
Then we can selectively display charts
Explain the screen in one sentence, or break it down

44. Improvement

45. Virtuous cycle
Measure
Model
ExecuteMeasure
Compare

46. Virtuous cycle
Measure
Model
ExecuteMeasure
Compare
PRODUCTION
PERF ENV

47. Virtuous cycle
Measure
Model
ExecuteMeasure
Compare
Use the same
tooling
Track
divergence

48. Regression tests
If we find a performance issue, try to add a test
that demonstrates the problem
This helps in the investigation phase, and
ensures regressions do not occur
Be careful with assertions

49. In a nutshell...

50. Key points
Use a known-good framework if possible
If you have to roll your own: peer review, measure it,
understand it
Data volume can be oppressive, use or develop tooling to
understand results
Test with realistic data/load distribution

51. Key points
Are we confident that our performance
testing will catch regressions before they
make it to production?

52. Thank you!
● @epickrram
● https://epickrram.blogspot.com
● recruitment@improbable.io

53. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
cd-performance-testing