Выступление на конференции DUMP-2016.

1. Профилирование
распределенных систем
Александр Казаков
СКБ Контур

3. 2000 компонентов,
взаимодействующих по сети

4. WEB.APP
SQL
2005

5. WEB.APP
SQL
2005
logging

6. WEB.APP
SQL
logging
tracing
2005

7. WEB.APP
SQL
2016
WEB.APP
WEB.APP
SRV 1
SRV 1
SRV 1
SRV 4
SRV 4
SRV 4
SRV 2
SRV 2
SRV 2
DATA 99
DATA 2
DATA 1
SRV 3
SRV 3
SRV 3
SRV 5
SRV 5
SRV 5

8. WEB.APP
SQL
WEB.APP
WEB.APP
SRV 1
SRV 1
SRV 1
SRV 4
SRV 4
SRV 4
SRV 2
SRV 2
SRV 2
DATA 99
DATA 2
DATA 1
SRV 3
SRV 3
SRV 3
SRV 5
SRV 5
SRV 5
2016

9. WEB.APP
SQL
WEB.APP
WEB.APP
SRV 1
SRV 1
SRV 1
SRV 4
SRV 4
SRV 4
SRV 2
SRV 2
SRV 2
DATA 99
DATA 2
DATA 1
SRV 3
SRV 3
SRV 3
SRV 5
SRV 5
SRV 5
2016

10. WEB.APP
SQL
WEB.APP
WEB.APP
SRV 1
SRV 1
SRV 1
SRV 4
SRV 4
SRV 4
SRV 2
SRV 2
SRV 2
DATA 99
DATA 2
DATA 1
SRV 3
SRV 3
SRV 3
SRV 5
SRV 5
SRV 5
2016

11. WEB.APP
SQL
WEB.APP
WEB.APP
SRV 1
SRV 1
SRV 1
SRV 4
SRV 4
SRV 4
SRV 2
SRV 2
SRV 2
DATA 99
DATA 2
DATA 1
SRV 3
SRV 3
SRV 3
SRV 5
SRV 5
SRV 5
300 СЕРВИСОВ
2000 КОМПОНЕНТОВ
2016

13. >ls -sh log2016.03.23
3.9G log2016.03.23

14. 2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [46556d43] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [46556d43] PageAsyncTask registered: Kontur.MR.WebApp.MergedIndicesTask`1[Kontur.M
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [6d688c14] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [6d688c14] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask started >>>
2016-03-23 09:57:54,131 96925771 INFO [T-b7adbebc(-)] [12472599] OrganizationsCache. Cache hit Organizations.DBS-HECTOR-MCHECTOR.reg_regusers.536
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask onSuccess started
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] OrganizationsGetSuccess: CurrentOrganization = '849073b7-a9cc-4428-819a-89c07bba44
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [3ab9c866] ConcurrentAsyncTaskManager waiting for all tasks to finish
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask onSuccess started
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] ReportsFilterState: UserId: 520fcedc-6a39-47b5-82e7-a5e335347896, OrgId: 849073b7-a9
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] Reports.Count: '10', response.ItemsRemainInIndex: '5'
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 INFO [T-b7adbebc(-)] [12472599] [ZebraSearch-124205838960] Success. Time = 00:00:00.0018612; Count = 0; Tablets polled
2016-03-23 09:57:54,131 96925771 INFO [T-be97f4ce(-)] [86468670] [124205838953] FileDownloader. Successfully performed lazy cache update.
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838953] IntelligentChunkDownloader. Managed to read 8700 bytes from 192.168.147.
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] FileDownloader. Started downloading file mr/reports/work/content2 from offs
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] IntelligentChunkDownloader. Requested to read chunk fdde5bbc-dc3f-4de8-
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] FileDownloader. Successfully performed lazy cache update.
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [73461e25] ConcurrentAsyncTaskManager waiting for all tasks to finish
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraCount-124205838961] Success. Time = 00:00:00.0164756; Count = 0; Tablets polled =
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [3ab9c866] ConcurrentAsyncTaskManager finished <<<<<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] ListRequest: Gf: -2147483648, PeriodEndLeft: 01.01.0001 0:00:00, PeriodEndRight: 31.12.
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraBatchSearch-124205838963] Tablet servers to poll:
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraBatchSearch-124205838963] Begin send request to TS 192.168.147.105:23673. Timeo

15. >grep "Success. Time =" log2016.03.23 > zebra.times
>wc -l zebra.times
957688 zebra.times

16. >grep "Success. Time =" log2016.03.23 > zebra.times
>wc -l zebra.times
957688 zebra.times
>grep "Success. Time = 00:00:0[1-9]" zebra.times > slow.reqs
>wc -l slow.reqs
2482 slow.reqs

17. >grep "Success. Time =" log2016.03.23 > zebra.times
>wc -l zebra.times
957688 zebra.times
>grep "Success. Time = 00:00:0[1-9]" zebra.times > slow.reqs
>wc -l slow.reqs
2482 slow.reqs
>grep -o "2016-03-23 ..:." slow.reqs | uniq -c > slow.reqs.rate
41 2016-03-23 08:0
10 2016-03-23 08:1
12 2016-03-23 08:2
49 2016-03-23 08:3
31 2016-03-23 08:4
52 2016-03-23 08:5
133 2016-03-23 09:0
81 2016-03-23 09:1
65 2016-03-23 09:2
67 2016-03-23 09:3
555 2016-03-23 09:4
583 2016-03-23 09:5

18. [ZebraSearch-124205847104] Success. Time = 00:00:01.0586813;

19. 2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [46556d43] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [46556d43] PageAsyncTask registered: Kontur.MR.WebApp.MergedIndicesTask`1[Kontur.M
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [6d688c14] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [6d688c14] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask started >>>
2016-03-23 09:57:54,131 96925771 INFO [T-b7adbebc(-)] [12472599] OrganizationsCache. Cache hit Organizations.DBS-HECTOR-MCHECTOR.reg_regusers.536
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask onSuccess started
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] OrganizationsGetSuccess: CurrentOrganization = '849073b7-a9cc-4428-819a-89c07bba44
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [33c6b348] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [3ab9c866] ConcurrentAsyncTaskManager waiting for all tasks to finish
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask onSuccess started
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] ReportsFilterState: UserId: 520fcedc-6a39-47b5-82e7-a5e335347896, OrgId: 849073b7-a9
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] Reports.Count: '10', response.ItemsRemainInIndex: '5'
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,131 96925771 DEBUG [T-b7adbebc(-)] [12472599] [04c0c9f9] PageAsyncTask completed <<<
2016-03-23 09:57:54,131 96925771 INFO [T-b7adbebc(-)] [12472599] [ZebraSearch-124205838960] Success. Time = 00:00:00.0018612; Count = 0; Tablets polled
2016-03-23 09:57:54,131 96925771 INFO [T-be97f4ce(-)] [86468670] [124205838953] FileDownloader. Successfully performed lazy cache update.
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838953] IntelligentChunkDownloader. Managed to read 8700 bytes from 192.168.147.
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] FileDownloader. Started downloading file mr/reports/work/content2 from offs
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] IntelligentChunkDownloader. Requested to read chunk fdde5bbc-dc3f-4de8-
2016-03-23 09:57:54,147 96925786 INFO [T-be97f4ce(-)] [86468670] [124205838962] FileDownloader. Successfully performed lazy cache update.
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [5ed38ee2] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [73461e25] ConcurrentAsyncTaskManager waiting for all tasks to finish
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [007cfc91] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-3d96d1c5(-)] [27914425] [534d5161] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraCount-124205838961] Success. Time = 00:00:00.0164756; Count = 0; Tablets polled =
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask onSuccess started
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask onSuccess completed
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [5eb39b03] PageAsyncTask completed <<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] [3ab9c866] ConcurrentAsyncTaskManager finished <<<<<<
2016-03-23 09:57:54,147 96925786 DEBUG [T-b7adbebc(-)] [12472599] ListRequest: Gf: -2147483648, PeriodEndLeft: 01.01.0001 0:00:00, PeriodEndRight: 31.12.
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraBatchSearch-124205838963] Tablet servers to poll:
2016-03-23 09:57:54,147 96925786 INFO [T-b7adbebc(-)] [12472599] [ZebraBatchSearch-124205838963] Begin send request to TS 192.168.147.105:23673. Timeo

20. 15 минут, чтобы подогнать под ответ

21. а как делают другие?

22. ELK

23. ELK

25. Dapper, a Large-Scale
Distributed Systems
Tracing Infrastructure

26. Google Dapper
trace
WEB.APP
WEB.APP
WEB.APP
SRV 2
SRV 2
SRV 2
SRV 3
SRV 3
SRV 3
SRV 4
SRV 4
SRV4

27. span
Google Dapper

28. span
• client send
• server receive
• server send
• client receive
Google Dapper

29. Google Dapper

30. Google Dapper

31. Twitter Zipkin
(Big Brother Bird)

32. Twitter Zipkin

33. Twitter Zipkin
Zipkin

34. Java
Zipkin
Twitter Zipkin

35. https://github.com/ditrace

37. RESTful API
C# client

38. C# client
https://github.com/ditrace/csharp

39. C# client
https://github.com/ditrace/csharp
1. Реализует отправку данных в DiTrace

40. C# client
https://github.com/ditrace/csharp
1. Реализует отправку данных в DiTrace
2. Помогает прокинуть контекст между
клиентом и сервером

41. C# client
https://github.com/ditrace/csharp
1. Реализует отправку данных в DiTrace
2. Помогает прокинуть контекст между
клиентом и сервером
3. Хранит текущий контекст в CallContext

42. Logging
CallContext хранит контекст трассировки
DEBUG [T-b7adbebc(-)] [04c0c9f9] PageAsyncTask onSuccess completed
DEBUG [T-b7adbebc(-)] [04c0c9f9] PageAsyncTask completed <<<
DEBUG [T-3d96d1c5(-)] [5ed38ee2] PageAsyncTask onSuccess started
DEBUG [T-3d96d1c5(-)] [5ed38ee2] PageAsyncTask onSuccess completed
DEBUG [T-3d96d1c5(-)] [5ed38ee2] PageAsyncTask completed <<<
DEBUG [T-3d96d1c5(-)] [73461e25] ConcurrentAsyncTaskManager waiting for all tasks to finish

44. Use case 1

52. Use case 2

54. T-b02385ac(+)

58. VIEW LOGS

61. DOWNLOAD

62. Use case 3

64. http://ditrace.readthedocs.org
https://www.elastic.co
http://research.google.com/pubs/pub36356.html
http://zipkin.io

65. Спасибо!
Александр Казаков
kazakov.dev@gmail.com