Everyone wants to understand what their application is really doing in production, but this information is normally invisible to developers. Profilers tell you what code your application is running but few developers profile and mostly on their development environments. Thankfully production profiling is now a practical reality that can help you solve and avoid performance problems. Profiling in development can be problematic because it’s rare that you have a realistic workload or performance test for your system. Even if you’ve got accurate performance tests maintaining these and validating that they represent production systems is hugely time consuming and hard. Not only that but often the hardware and operating system that you run in production are different from your development environment. This pragmatic talk will help you understand the ins and outs of profiling in a production system. You’ll learn about different techniques and approaches that help you understand what’s really happening with your system. This helps you to solve new performance problems, regressions and undertake capacity planning exercises. Find out how profiling in production can uncover performance bottlenecks, aid scalability and reduce your costs.

1. Continuous Profiling in
Production: What, Why and How
Richard Warburton (@richardwarburto)
Sadiq Jaffer (@sadiqj)
https://www.opsian.com

2. Why Performance Tools Matter
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

3. Known Knowns

4. Known Unknowns

5. Unknown Unknowns

6. Why Performance Tools Matter
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

7. Development isn’t Production
Performance testing in development can be easier
May not have access to production
Tooling often desktop-based
Not representative of production

8. Unrepresentative Hardware
vs

9. Unrepresentative Software

10. Unrepresentative Workloads
vs

11. The JVM may have very different behaviour in production
Hotspot does adaptive optimisation
Production may optimise differently

13. Why Performance Tools Matter
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

14. Ambient/Passive/System Metrics
Preconfigured Numerical Measure
CPU Time Usage / Page-load Times
Cheap and sometimes effective

16. Logging
Records arbitrary events emitted by the system being monitored
log4j/slf4j/logback
Logs of GC events
Often manual, aids system understanding, expensive

17. Coarse Grained Instrumentation
Measures time within some instrumented section of the code
Time spent inside the controller layer of your web-app or performing SQL queries
More detailed and actionable though expensive

18. Production Profiling
What methods use up CPU time?
What lines of code allocate the most objects?
Where are your CPU Cache misses coming from?
Automatic, can be cheap but often isn’t

19. Where Instrumentation can be blind in the Real World
Problem: every 5 seconds an HTTP endpoint would be really slow.
Instrumentation: on the servlet request, didn’t even show the pause!
Cause: Tomcat expired its resources cache every 5 seconds, on load one resource
scanned the entire classpath

21. Surely a better way?
Not just Metrics - Actionable Insights
Diagnostics aren’t Diagnosis
What about Profiling?

22. Why Performance Tools Matter
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

23. How to use Production Profilers
1) Extract relevant time period and apps/machines
2) Choose a type of profile: CPU Time/Wallclock Time/Memory
3) View results to tell you what the dominant consumer of a resource is
4) Fix biggest bottleneck
5) Deploy / Iterate

24. CPU Time vs Wallclock Time

25. You need both CPU Time and Wallclock Time
CPU - Diagnose expensive computational hotspots and inefficient algorithms
Spot code that should not be executing but is ...
Wallclock - Diagnose blocking that stops CPU usage
e.g blocking on external IO and lock contention issues

26. Profiling Hotspots

27. Profiling Treeviews

28. Profiling Flamegraphs

29. Instrumenting Profilers
Add instructions to collect timings (Eg: JVisualVM Profiler)
Inaccurate - modifies the behaviour of the program
High Overhead - > 2x slower

30. Sampling/Statistical Profilers
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml() ??? ???

31. Safepoint Bias after Inlining
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml() ??? ???

32. Time to Safepoint
-XX:+PrintSafepointStatistics
Threads
Safepoint poll
VMOperation

33. Statistical Profiling in Java
Problem: getAllStackTraces is expensive to do frequently and inaccurate, also only
gives us Wallclock time
Need ways to:
1. Interrupt application
2. Sample resource of interest

34. Advanced Statistical Profiling in Java
● Interrupt with OS signals
○ Delivered to handler on only one thread
○ Lightweight
● Sample resource of interest
○ Use AsyncGetCallTrace to sample stack
○ Examine JVM internals for other resources

35. Advanced Statistical Profiling in Java
Approach not used by existing profilers (VisualVM and desktop commercial
alternatives)
Can give very low overheads (<1%) for reasonable sampling rates

36. People are put off by practical as
much as technical issues

37. Barriers to Ad-Hoc Production Profiling
Generally requires access to
production
Process involves manual work - hard
to automate
Low-overhead open source profilers
without commercial support

38. What if we profiled all the time?

39. Historical Data
Allows for post-hoc incident analysis
Enables correlation with other data/metrics
Performance regression analysis

40. Putting Samples in Context
Application version
Environment parameters (machine type, CPU, location, etc.)
Ad-hoc profiling we can’t do this

41. Opsian - Continuous Profiling
Opsian
Aggregation
service
Web Reports
JVM Agents

42. Opsian - Continuous Profiling
Opsian
Aggregation
service
Web Reports
JVM Agents
Lightweight agents commercially-supported
and validated on wide range of JVM Versions
(~6 to 11)
WallClock and CPU Time

43. Opsian - Continuous Profiling
Opsian
Aggregation
service
Web Reports
JVM Agents
Custom-built database capable of
continuously aggregating, indexing and
compressing raw profiling samples
across server infrastructure
Fast real-time reports over arbitrary
dimensions

44. Opsian - Continuous Profiling
Opsian
Aggregation
service
Web Reports
JVM Agents
Browser-based shareadable Hot Spots,
Flame Graphs, Tree Views and Version
Comparison reports

45. Summary
It’s possible to profile in production with low overhead
To overcome practical issues we can profile production all the time
We gain new capabilities by profiling all the time

46. Why Performance Tools Matter
Development isn’t Production
Profiling vs Monitoring
Production Profiling
Conclusion

47. Performance Matters
Development isn’t Production
Metrics can be unactionable
Instrumentation has high overhead
Continuous Profiling provides insight

48. We need an attitude shift on profiling
+ monitoring

49. ContinuousProactive
not Reactive
Systematic
not Ad Hoc

50. Please do Production Profiling.
All the time.