Goal of this talk is to get through performance analysis process of simple calculator application implemented as a set of microservices with following FOSS toolset: k6 and gatling for traffic generation, prometheus for monitoring, perf Linux profiler and FlameGraph project for flame graphs generation.

1. Performance testing of
microservices
IN ACTION
Alexander Kachur
Lead QA Automation Engineer
at Waverley Software
alexander.kachur[at]gmail.com
Julien Dupre

2. 1. Consideration points on picking load generator
2. How to use CPU flame graph to identify bottleneck
3. Issues with the Cloud and Docker
4. How to use perf to figure out what happened
5. Links
Take Aways

6. Containers

8. Simple Queueing Model
Service
center
Response
time
Input
Output
Queue

12. P is for Profiling

13. Service Map
Jeremy Cook

16. Reporting

17. Reporting

21. Round 1

22. Round 1

23. Flame Graphs
main()
create()find()
query() init() print()
main()
find()
main()
init()
create()
main()
create()
query() print()

24. Flame Graphs

25. Flame Graphs

26. Round 1
DNS

27. Round 2

28. Round 2

29. Flame Graphs

30. Flame Graphs

31. Flame Graphs

32. Round 2
TTY stdout

33. Round 3

34. Round 3

35. Flame Graphs

36. Flame Graphs

37. Flame Graphs

38. Round 3
keep-alive

39. Final Round

40. Final Round

41. Flame Graphs with Docker
0. [sudo sysctl kernel.perf_event_paranoid=0]
1. perf record -F 99 -g -p ${PID}
2. docker exec -it ${container_ID} top
3. docker exec ${container_ID} cat /tmp/perf-${internal_PID}.map > /tmp/perf-
${PID}.map
4. perf script -f > out.perf
5. ./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
6. ./FlameGraph/flamegraph.pl --hash out.folded > out.svg

42. Flame Graphs with Docker
0. [sudo sysctl kernel.perf_event_paranoid=0]
1. perf record -F 99 -g -p ${PID}
2. docker exec -it ${container_ID} top
3. docker exec ${container_ID} cat /tmp/perf-${internal_PID}.map > /tmp/perf-
${PID}.map
4. perf script -f > out.perf
5. ./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
6. ./FlameGraph/flamegraph.pl --hash out.folded > out.svg

43. Flame Graphs with Docker
0. [sudo sysctl kernel.perf_event_paranoid=0]
1. perf record -F 99 -g -p ${PID}
2. docker exec -it ${container_ID} top
3. docker exec ${container_ID} cat /tmp/perf-${internal_PID}.map > /tmp/perf-
${PID}.map
4. perf script -f > out.perf
5. ./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
6. ./FlameGraph/flamegraph.pl --hash out.folded > out.svg

44. Flame Graphs with Docker
0. [sudo sysctl kernel.perf_event_paranoid=0]
1. perf record -F 99 -g -p ${PID}
2. docker exec -it ${container_ID} top
3. docker exec ${container_ID} cat /tmp/perf-${internal_PID}.map > /tmp/perf-$
{PID}.map
OR
4. perf script | perl ./FlameGraph/stackcollapse-perf.pl |gzip | curl --data-binary
@- -H "Content-Tye: gzip" -H "Accept-Encoding: gzip" flamegraph-
generator.herokuapp.com/api/generate
https://github.com/olka/flamegraph-generator

45. RPC problem

46. RPC solutions

47. Performance in the Cloud

48. Performance in the Cloud

49. Performance in the Cloud

50. Drill Down

51. Drill Down

52. Drill Down

53. Drill Down

54. Drill Down

55. Conclusion
1. Why?
2. Why?
3. Why?
4. Why?
5. Why?

56. Gurus
Brendan Gregg - brendangregg.com | FlameGraph
Baron Schwartz - xaprb.com
Rob Harrop - robharrop.github.io
Neil J. Gunther - perfdynamics.com/Manifesto/USLscalability.html
Aleksey Shipilëv- shipilev.net
Sasha Goldshtein - blog.sashag.net
Nitsan Wakart - psy-lob-saw.blogspot.co.il
Aleksey Lavrenyuk - events.yandex.ru/lib/people/302066
Vladimir Sitnikov- lektorium.tv/speaker/27401