Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

984 views

Published on

2018 아이펀팩토리 데브데이 서버위더스
(iFunFactory DevDay Server,Withus)

제목 : 코드 수정 없이 리눅스 게임 서버 성능 평가
발표자 : 김진욱 CTO
일정 : 2018년 03월 28일
개요 : 서버에 코드 추가 없이도 고수준에서 서버 성능 분석이 가능하다!
실행 중인 서버를 외부에서 관찰하여 최대한 성능을 분석하고 디버깅 하는 법을 공개 합니다.

아래링크를 통해 아이펀팩토리의 더 많은 정보를 얻으실 수 있습니다.
*아이펀팩토리 홈페이지 : https://ifunfactory.com/
*아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/

Published in: Technology

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

  1. 1. <jinuk.kim@ifunfactory.com>
  2. 2. 2018 iFunFactory Dev Day (2013-) • (C++, C#) • (Python) (2012-2013) • (Python) (2007-2012) • (C++) • (C++, C#, Python) • (C#)
  3. 3. 2018 iFunFactory Dev Day Contents. 01 02 03 04 05 06
  4. 4. 2018 iFunFactory Dev Day 01
  5. 5. 2018 iFunFactory Dev Day ? ? Linux eBPF + BCC
  6. 6. 2018 iFunFactory Dev Day : 02
  7. 7. 2018 iFunFactory Dev Day : • (2017 ) • + • • DB / • CPU • (lock) • (DB, redis, 3rd party API) •
  8. 8. 2018 iFunFactory Dev Day :
  9. 9. 2018 iFunFactory Dev Day : • / DB / CPU ( / / / ) • DB • /
  10. 10. 2018 iFunFactory Dev Day • . • / • / / (fiber) / • • / • •
  11. 11. 2018 iFunFactory Dev Day • / . • . • • • / /
  12. 12. 2018 iFunFactory Dev Day 03
  13. 13. 2018 iFunFactory Dev Day • /
  14. 14. 2018 iFunFactory Dev Day : CPU • GNU Compiler Collection (GCC) • -pg (gprof) • LLVM/Clang • -fprofile-instr-generate • Visual C++ instrumentation • Valgrind Callgrind
  15. 15. 2018 iFunFactory Dev Day • / • / • • /
  16. 16. 2018 iFunFactory Dev Day GProf : % cumulative self self total time seconds seconds calls ms/call ms/call name 14.29 0.03 0.03 207323 0.00 0.00 crow::json::escape 14.29 0.06 0.03 3059 0.01 0.02 crow::CrowApp::handle 9.52 0.08 0.02 2638 0.01 0.02 crow::json::dump_internal 4.76 0.09 0.01 307190 0.00 0.00 _Hashtable<…>::insert_unique 4.76 0.10 0.01 208356 0.00 0.00 _Hashtable<…>::_M_rehash 4.76 0.11 0.01 19129 0.00 0.00 vector<crow::json::wvalue>::_M_emplace_back_aux 4.76 0.12 0.01 11961 0.00 0.00 crow::Connection<crow::SocketAdaptor, crow::CrowApp>::cancel_deadline_timer
  17. 17. 2018 iFunFactory Dev Day GProf :
  18. 18. 2018 iFunFactory Dev Day • gprof: 30% - 300% • Valgrind: 400% - 4,000% • 3rd (.dll, .so) •
  19. 19. 2018 iFunFactory Dev Day . / • (CPU, , , ) • •
  20. 20. 2018 iFunFactory Dev Day • IO 10ms ? • ? • ?
  21. 21. 2018 iFunFactory Dev Day 04
  22. 22. 2018 iFunFactory Dev Day CPU • . • Linux perf (kernel v2.6+) • Linux eBPF (kernel v3.18+; v4.9+) • Google Performance Tools • Visual C++ : CPU
  23. 23. 2018 iFunFactory Dev Day • ( 50 - 200 ) • CPU • • /
  24. 24. 2018 iFunFactory Dev Day : CPU Linux v4.13 eBPF + BCC profile: Flamegraph: / 
 https://github.com/brendangregg/FlameGraph
  25. 25. 2018 iFunFactory Dev Day
  26. 26. 2018 iFunFactory Dev Day
  27. 27. 2018 iFunFactory Dev Day 
 OS / ( ), OS
  28. 28. 2018 iFunFactory Dev Day OS OS OS ! • Linux v4.13 randstruct • eBPF #define
  29. 29. 2018 iFunFactory Dev Day ? •Disk I/O ? •CPU ? • (lock) ? • API ? • ? • ?
  30. 30. 2018 iFunFactory Dev Day Linux •eBPF (kernel v3.18+; 4.9+ ) •perf (kernel v2.6+; ) Windows •Event Tracing for Windows (ETW; Win 8+) •Windows Event Logging ( ; Win 2k+)
  31. 31. 2018 iFunFactory Dev Day ? / ?
  32. 32. 2018 iFunFactory Dev Day 

  33. 33. 2018 iFunFactory Dev Day /
  34. 34. 2018 iFunFactory Dev Day Linux eBPF + BCC • eBPF: / • BCC: Python / lua / go eBPF
  35. 35. 2018 iFunFactory Dev Day eBPF OS 
 (x86-64/arm64 ) VM • / ( ) • • JIT
  36. 36. 2018 iFunFactory Dev Day eBPF: ? • • (= ) • ( ) • ( )
  37. 37. 2018 iFunFactory Dev Day BCC • eBFP Compiler Collection • Python / lua / go / • C eBPF • eBPF • python / lua / go
  38. 38. 2018 iFunFactory Dev Day eBPF + BCC: CPU, , I/O, eBPF + BCC ( ) : 1. , 2. eBPF + BCC 3.
  39. 39. 2018 iFunFactory Dev Day BCC : BCC funclatency /
  40. 40. 2018 iFunFactory Dev Day : eBPF (1)
  41. 41. 2018 iFunFactory Dev Day : eBPF (2)
  42. 42. 2018 iFunFactory Dev Day
  43. 43. 2018 iFunFactory Dev Day : BCC + BPF
  44. 44. 2018 iFunFactory Dev Day :
  45. 45. 2018 iFunFactory Dev Day Linux Kernel v4.13+ randstruct . BCC “bpf_text” #define randomized_struct_fields_start struct { #define randomized_struct_fields_end };
  46. 46. 2018 iFunFactory Dev Day 05 HTTP API
  47. 47. 2018 iFunFactory Dev Day HTTP API I/O, DB , ( ) /OS : flaskr • Crow HTTP • CrowDB + MariaDB • RESTful API • https://github.com/jinukkim/devday-2018-03
  48. 48. 2018 iFunFactory Dev Day HTTP : URL / HTTP HTTP ,
 4 

  49. 49. 2018 iFunFactory Dev Day HTTP : DB JSON HTTP
  50. 50. 2018 iFunFactory Dev Day HTTP : MariaDB DB HTTP DNS
  51. 51. 2018 iFunFactory Dev Day 4 1000 250 ≦ 4 ms
  52. 52. 2018 iFunFactory Dev Day siege: HTTP siege -c 32
 -r 16384
 -b
 http://example.com URL 32 (-c32) (-b) 16k (-r 16384)
  53. 53. 2018 iFunFactory Dev Day : 110 / : 27.5 / : 36 ms
  54. 54. 2018 iFunFactory Dev Day ?
  55. 55. 2018 iFunFactory Dev Day #1: CPU
  56. 56. 2018 iFunFactory Dev Day : CPU ? eBPF + BCC: profile • • 30 , 199 flaskr sudo profile -p $(pgrep -nx flaskr)
 -f
 -F 199 30

  57. 57. 2018 iFunFactory Dev Day (1): Flamegraph
  58. 58. 2018 iFunFactory Dev Day (2): DNS C (10%) • __GI_nss_files_servent (OS ) • TCP (6.3%) • (3%) • (2%) ( ) !
  59. 59. 2018 iFunFactory Dev Day CPU ? Linux htop (= top ) • CPU 42% CPU .
  60. 60. 2018 iFunFactory Dev Day #2:
  61. 61. 2018 iFunFactory Dev Day #2 ? HTTP API : • DB • SQL • SQL • etc.
  62. 62. 2018 iFunFactory Dev Day : DB ? eBPF + BCC: mysql_qslower: eBPF + BCC: funclatency • MariaDB (mysql_ )
 sudo funclatency.py -p $(pgrep -nx flaskr)
 -F /usr/lib/mariadb/libmariadb.so:mysql_*
  63. 63. 2018 iFunFactory Dev Day : SQL • : 0.5 ms - 2 ms • 2 ms ≪ 36 ms ( )
  64. 64. 2018 iFunFactory Dev Day : SQL • : 0.008 ms - 0.032 ms • 0.032 ms ≪ 36 ms ( )
  65. 65. 2018 iFunFactory Dev Day
  66. 66. 2018 iFunFactory Dev Day #3: DB
  67. 67. 2018 iFunFactory Dev Day : ? ? • CPU • OS CPU : • OS CPU • CPU =
  68. 68. 2018 iFunFactory Dev Day : eBPF + BCC: offcputime • •
  69. 69. 2018 iFunFactory Dev Day : Flamegraph • poll() : C . 78%; • epoll() : . 16%; boost::asio
  70. 70. 2018 iFunFactory Dev Day : poll() ?
  71. 71. 2018 iFunFactory Dev Day poll() C++ ( ) !
  72. 72. 2018 iFunFactory Dev Day ? • • : • • /
  73. 73. 2018 iFunFactory Dev Day eBPF + BCC: offwaketime : • • offwaketime -p $(pgrep -nx flaskr) -f
  74. 74. 2018 iFunFactory Dev Day • UDP/IP poll
  75. 75. 2018 iFunFactory Dev Day : UDP ?
  76. 76. 2018 iFunFactory Dev Day UDP ? HTTP TCP/IP UDP : • DHCP • DNS • RTP • etc… DNS . ( )
  77. 77. 2018 iFunFactory Dev Day : DNS ?
  78. 78. 2018 iFunFactory Dev Day DNS DNS (C API ) • getnameinfo • getaddrinfo • gethostname • etc… eBPF + BCC: funclatency
  79. 79. 2018 iFunFactory Dev Day • getnameinfo • 32ms - 134ms • (36ms)
  80. 80. 2018 iFunFactory Dev Day : DNS DNS DNS dnsmasq : 1,567 / • 390 / , 2.5 ms : 14 • : 110 / vs. 1,567 /
  81. 81. 2018 iFunFactory Dev Day : MS Windows + ETW GUI + / ETW / Xperf : The Lost Xperf Documentation on CPU Scheduling
  82. 82. 2018 iFunFactory Dev Day 06 
 /
  83. 83. 2018 iFunFactory Dev Day (runqlat) • ( ?) Disk I/O • I/O (fileslower) • (filetop) • ext4slower, btrfsslower,
  84. 84. 2018 iFunFactory Dev Day (memleak) • 30 ? (deadlock_detector) • mutex
  85. 85. 2018 iFunFactory Dev Day Android / https://lwn.net/Articles/742363/
  86. 86. 2018 iFunFactory Dev Day Android / eBPF x86-64 / arm64 1. 
 + LLVM 2. Android 3. 
 4. 
 / / https://github.com/joelagnel/bpfd
  87. 87. 2018 iFunFactory Dev Day Android / BPFd 
 (https://github.com/joelagnel/bpfd) eBPF + BCC (v4.6+; v4.9+) • Android 8.0 Oreo v4.4 AOSP 8.0 v4.10;
  88. 88. 2018 iFunFactory Dev Day 
 
 eBPF + BCC
  89. 89. Q&A 2018 iFunFactory Dev Day 2018 iFunFactory Dev Day Thank You! 660, 1 B 606 +82-70-4923-6566 www.ifunfactory.cominfo@ifunfactory.com

×