This session will take us from theory to actual practice demonstrating what continuous profiling is. By the end of this talk, every attendee will learn how to understand and optimize their code performance even in the most complex production environments. With the increasing complexity of modern applications, continuous profiling methods and tools are gaining popularity among the Developer and Engineering communities. In this session, we cover what continuous profiling entails and why you should implement a profiler into your tech stack (if you have not done so already). We will then bring theory to practice and demonstrate a real-life scenario using gProfiler, a free open-source continuous profiling tool, covering Linux servers on multiple architectures (such as Graviton).

1. Optimize Performance with
Continuous Production Profiling
Yishai Zinkin
Senior Software Engineer @ Granulate

2. Why optimize?
● In most cases in software - faster is better
● Reduce computing cost
Who cares about profiling anyway?
Your CPU is one of your most valuable resources.
You want to know what it spends time on!

3. Source: https://www.shopify.com/enterprise/site-performance-page-speed-ecommerce

4. And why profile?

5. What is Profiling?
● The dynamic inspection of running programs
● Most commonly, used to aid optimization
○ Can also be used to aid debugging
● Many types (CPU, memory, networking/IO, locks…)
Today’s focus: CPU Profiling

6. ● Profile what you want to optimize
○ If you profile test load and test data you might end
up optimizing the tests instead of your production
What and when should you profile?
Continuous Production Profiling
The key to accurate optimizations is profiling the
production environment continuously

8. Continuous Profiling
Profile
Analyze results
Implement code
improvements
Deploy

9. ● Each short period of time we take a “snapshot” of our
program - a stack trace
● If we see a function at the top of the stack in e.g. 30% of
the samples - it (probably) consumes 30% of our CPU
Samples Profiling
The solution for Profiling in production

10. Flame Graphs
Stack traces visualization
a()
b()
c()
1 3
a()
b()
a()
b()
c()
g()
f()
5 6 7 9
e()
h()
10
d()
a()
11
d()
a() a()
8
b()
c()
a()
b()
c()
g()
f()
a()
b()
c()
g()
d()
a()
a()
4
f()
d()
a()
2
e()

11. Flame Graphs
Stack traces visualization
1 2 4
3 5 6
7 8
9 10
11
a()
f()
e()
h()
d()
a()
d()
a()
d()
a()
d()
a()
e()
a()
b()
c()
g()
a()
b()
c()
g()
f() f()
a()
b()
a()
b()
c()
a()
b()
c()
a()
b()
c()
g()

12. Flame Graphs
Stack traces visualization
f()
h()
d()
e()
f()
a()
b()
c()
g()

13. A simple example
Flame Graphs
def sum_10M():
return sum(range(10**7))
def sum_100M():
return sum(range(10**8))
def sleepy():
sleep(1)
def main():
sum_10M()
sleepy()
sum_100M()
if __name__ == "__main__":
main()

14. py-spy record --function --gil --nonblocking -- python3 example.py
A simple example
Flame Graphs

15. Samples

16. Live Demo

20. Advanced Techniques

22. #1 Weights
● Profiling & optimizations are about
the big problems
● Let’s kick away the smaller ones!

23. #1 Weights

24. #1 Weights

25. #2 Table view
● Aggregate by function (and not by stack)
● Sort by different fields

26. #2 Table view
● Aggregate by function (and not by stack)
b: occurrences == 3
d: occurrences == 1
a
b
b
d
a
c
b

27. #2 Table view
● Sort by different fields
a: totaltime == 100%, owntime == 25%
b: totaltime == 30%, owntime == 30%
d: totaltime == 20%, owntime == 10%
a
b
b
d
a
c
b

28. #3 Split view
● Aggregate different callstacks that include a particular function

29. #3 Split view
● Aggregate different callstacks that include a particular function

30. Key Takeaways
● Profiling is an extremely powerful tool that can help us to
provide effective improvements to large code bases easily
○ Don’t optimize prematurely!
● Profile what you want to optimize:
○ Production environment
○ Continuously
Just start profiling :)

31. Q&A

32. Thank you and happy profiling!
To start free profiling visit gProfiler.io or scan the QR code:
Feel free to reach me out at yishai.zinkin@granulate.io