This document discusses production profiling, including why it is important, how it differs from development and monitoring, and how to do it effectively. Production profiling provides deep insights into performance issues by measuring things like CPU time usage and memory allocation at the method/line level, unlike coarse ambient metrics. It helps identify specific bottlenecks to fix. Historical profiling data allows post-hoc analysis and correlation with other metrics. New low-overhead sampling profilers make always-on profiling possible, unlike traditional high-overhead instrumentation. The key is taking a proactive, systematic approach to profiling in production environments.

1. Production Profiling: What, Why
and How
Richard Warburton (@richardwarburto)
https://www.opsian.com

2. Why Performance Matters
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

3. Customer Experience

4. Amazon: 100ms of latency costs 1% of sales
Google: 500ms seconds in search page generation time drops traffic by 20%
Responsive Applications make more Money

5. Reduce Costs

6. Why Performance Matters
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

7. Development isn’t Production
Testing in development can be easier
Tooling often desktop-based
Not representative of production

8. Unrepresentative Hardware
vs

9. Unrepresentative Software

10. Unrepresentative Workloads
vs

11. The JVM may have very different behaviour in production
Hotspot does adaptive optimisation
Production may optimise differently

13. Why Performance Matters
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

14. Ambient/Passive/System Metrics
Preconfigured numerical measure about the
system
CPU Time Usage / Page-load Times
Cheap and sometimes effective

15. Coarse Grained Instrumentation
Measures time within some instrumented section of the code
Time spent inside the controller layer of your web-app or performing SQL queries
More detailed and actionable though expensive

16. Production Profiling
What methods use up CPU time?
What lines of code allocate the most objects?
Where are your CPU Cache misses coming from?
Automatic, can be cheap but often isn’t

18. Understand don’t Overwhelm
Too Little
You can’t understand production
problems
Too Much
Needle in a Haystack
You are the problem (overhead)

19. Unactionable Metrics
Many metrics provide pretty graphs but don’t progress treatment

20. Normalisation of Deviance
“Some of the tests always fail, so we just ignore them.”
“Some of the alerts get triggered regularly, so we just ignore them.”
Alert false positives have a cost

21. Surely a better way?
Not just Metrics - Actionable Insights
What about Profiling?

22. Why Performance Matters
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

23. How to use Production Profilers
1) Extract relevant time period and apps/machines
2) Choose a type of profile: CPU Time/Wallclock Time/Memory
3) View results to tell you what the dominant consumer of a resource is
4) Fix biggest bottleneck
5) Deploy / Iterate

24. CPU Time vs Wallclock Time

25. Profiling Hotspots

26. Profiling Treeviews

27. Profiling Flamegraphs

28. Instrumenting Profilers
Add instructions to collect timings (Eg: JVisualVM Profiler)
Inaccurate - modifies the behaviour of the program
High Overhead - > 2x slower

29. Sampling/Statistical Profilers
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml() ??? ???

30. Safepoint Bias + Latency Impact
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml() ???

31. Advanced Statistical Profiling in Java
OS Signals to interrupt threads on resource consumption threshold
JVM’s signal handler-safe AsyncGetCallTrace to walk the stack

32. Open Source Java Profilers
High Overhead
VisualVM
hprof
Twitter’s CPUProfile
Anything GetAllStackTraces based
Low Overhead
Async Profiler
Honest Profiler
Java Mission Control

33. Profiling Support in the Linux Kernel
perf and eBPF
perf-map-agent for the JVM
Hardware events (L1/L2/L3 cache misses, branch mispredictions, etc.)

34. People are put off by practical as
well as technical issues

35. Barriers to Ad-Hoc Production Profiling
Low overhead unbiased tools are
relatively new
Generally requires access to
production
Process involves manual work - hard
to automate

36. What if we profiled all the time?

37. Historical Data
Allows for post-hoc incident analysis
Enables correlation with other data/metrics
Performance regression analysis

38. Aggregation
Aggregation and
Indexing
Samples
Reports

39. Putting Samples in Context
Application version
Environment parameters (machine type, CPU, location, etc.)
Desktop profilers don’t do this

40. Summary
Production Profiling is possible - all the time with low overhead
Production Profiling is desirable - can give deep insights
Problems can be overcome

41. Why Performance Matters
Development isn’t Production
Profiling vs Monitoring
Prod Profiling Problems
Conclusion

42. Performance Matters
Development isn’t Production
Metrics can be unactionable
Instrumentation has high overhead
Production Profiling provides insight

43. We need an attitude shift on profiling
+ monitoring

44. Always
On
Proactive
not Reactive
Systematic
not Ad Hoc

45. Please do Production Profiling.
All the time.

46. Any Questions?
https://www.opsian.com/

47. The End