Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

1,413 views

Published on

2018 아이펀팩토리 데브데이 서버위더스
(iFunFactory DevDay Server,Withus)

제목 : 코드 수정 없이 리눅스 게임 서버 성능 평가
발표자 : 김진욱 CTO
일정 : 2018년 03월 28일
개요 : 서버에 코드 추가 없이도 고수준에서 서버 성능 분석이 가능하다!
실행 중인 서버를 외부에서 관찰하여 최대한 성능을 분석하고 디버깅 하는 법을 공개 합니다.

아래링크를 통해 아이펀팩토리의 더 많은 정보를 얻으실 수 있습니다.
*아이펀팩토리 홈페이지 : https://ifunfactory.com/
*아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

  1. 1. <jinuk.kim@ifunfactory.com>
  2. 2. 2018 iFunFactory Dev Day (2013-) • (C++, C#) • (Python) (2012-2013) • (Python) (2007-2012) • (C++) • (C++, C#, Python) • (C#)
  3. 3. 2018 iFunFactory Dev Day Contents. 01 02 03 04 05 06
  4. 4. 2018 iFunFactory Dev Day 01
  5. 5. 2018 iFunFactory Dev Day ? ? Linux eBPF + BCC
  6. 6. 2018 iFunFactory Dev Day : 02
  7. 7. 2018 iFunFactory Dev Day : • (2017 ) • + • • DB / • CPU • (lock) • (DB, redis, 3rd party API) •
  8. 8. 2018 iFunFactory Dev Day :
  9. 9. 2018 iFunFactory Dev Day : • / DB / CPU ( / / / ) • DB • /
  10. 10. 2018 iFunFactory Dev Day • . • / • / / (fiber) / • • / • •
  11. 11. 2018 iFunFactory Dev Day • / . • . • • • / /
  12. 12. 2018 iFunFactory Dev Day 03
  13. 13. 2018 iFunFactory Dev Day • /
  14. 14. 2018 iFunFactory Dev Day : CPU • GNU Compiler Collection (GCC) • -pg (gprof) • LLVM/Clang • -fprofile-instr-generate • Visual C++ instrumentation • Valgrind Callgrind
  15. 15. 2018 iFunFactory Dev Day • / • / • • /
  16. 16. 2018 iFunFactory Dev Day GProf : % cumulative self self total time seconds seconds calls ms/call ms/call name 14.29 0.03 0.03 207323 0.00 0.00 crow::json::escape 14.29 0.06 0.03 3059 0.01 0.02 crow::CrowApp::handle 9.52 0.08 0.02 2638 0.01 0.02 crow::json::dump_internal 4.76 0.09 0.01 307190 0.00 0.00 _Hashtable<…>::insert_unique 4.76 0.10 0.01 208356 0.00 0.00 _Hashtable<…>::_M_rehash 4.76 0.11 0.01 19129 0.00 0.00 vector<crow::json::wvalue>::_M_emplace_back_aux 4.76 0.12 0.01 11961 0.00 0.00 crow::Connection<crow::SocketAdaptor, crow::CrowApp>::cancel_deadline_timer
  17. 17. 2018 iFunFactory Dev Day GProf :
  18. 18. 2018 iFunFactory Dev Day • gprof: 30% - 300% • Valgrind: 400% - 4,000% • 3rd (.dll, .so) •
  19. 19. 2018 iFunFactory Dev Day . / • (CPU, , , ) • •
  20. 20. 2018 iFunFactory Dev Day • IO 10ms ? • ? • ?
  21. 21. 2018 iFunFactory Dev Day 04
  22. 22. 2018 iFunFactory Dev Day CPU • . • Linux perf (kernel v2.6+) • Linux eBPF (kernel v3.18+; v4.9+) • Google Performance Tools • Visual C++ : CPU
  23. 23. 2018 iFunFactory Dev Day • ( 50 - 200 ) • CPU • • /
  24. 24. 2018 iFunFactory Dev Day : CPU Linux v4.13 eBPF + BCC profile: Flamegraph: / 
 https://github.com/brendangregg/FlameGraph
  25. 25. 2018 iFunFactory Dev Day
  26. 26. 2018 iFunFactory Dev Day
  27. 27. 2018 iFunFactory Dev Day 
 OS / ( ), OS
  28. 28. 2018 iFunFactory Dev Day OS OS OS ! • Linux v4.13 randstruct • eBPF #define
  29. 29. 2018 iFunFactory Dev Day ? •Disk I/O ? •CPU ? • (lock) ? • API ? • ? • ?
  30. 30. 2018 iFunFactory Dev Day Linux •eBPF (kernel v3.18+; 4.9+ ) •perf (kernel v2.6+; ) Windows •Event Tracing for Windows (ETW; Win 8+) •Windows Event Logging ( ; Win 2k+)
  31. 31. 2018 iFunFactory Dev Day ? / ?
  32. 32. 2018 iFunFactory Dev Day 

  33. 33. 2018 iFunFactory Dev Day /
  34. 34. 2018 iFunFactory Dev Day Linux eBPF + BCC • eBPF: / • BCC: Python / lua / go eBPF
  35. 35. 2018 iFunFactory Dev Day eBPF OS 
 (x86-64/arm64 ) VM • / ( ) • • JIT
  36. 36. 2018 iFunFactory Dev Day eBPF: ? • • (= ) • ( ) • ( )
  37. 37. 2018 iFunFactory Dev Day BCC • eBFP Compiler Collection • Python / lua / go / • C eBPF • eBPF • python / lua / go
  38. 38. 2018 iFunFactory Dev Day eBPF + BCC: CPU, , I/O, eBPF + BCC ( ) : 1. , 2. eBPF + BCC 3.
  39. 39. 2018 iFunFactory Dev Day BCC : BCC funclatency /
  40. 40. 2018 iFunFactory Dev Day : eBPF (1)
  41. 41. 2018 iFunFactory Dev Day : eBPF (2)
  42. 42. 2018 iFunFactory Dev Day
  43. 43. 2018 iFunFactory Dev Day : BCC + BPF
  44. 44. 2018 iFunFactory Dev Day :
  45. 45. 2018 iFunFactory Dev Day Linux Kernel v4.13+ randstruct . BCC “bpf_text” #define randomized_struct_fields_start struct { #define randomized_struct_fields_end };
  46. 46. 2018 iFunFactory Dev Day 05 HTTP API
  47. 47. 2018 iFunFactory Dev Day HTTP API I/O, DB , ( ) /OS : flaskr • Crow HTTP • CrowDB + MariaDB • RESTful API • https://github.com/jinukkim/devday-2018-03
  48. 48. 2018 iFunFactory Dev Day HTTP : URL / HTTP HTTP ,
 4 

  49. 49. 2018 iFunFactory Dev Day HTTP : DB JSON HTTP
  50. 50. 2018 iFunFactory Dev Day HTTP : MariaDB DB HTTP DNS
  51. 51. 2018 iFunFactory Dev Day 4 1000 250 ≦ 4 ms
  52. 52. 2018 iFunFactory Dev Day siege: HTTP siege -c 32
 -r 16384
 -b
 http://example.com URL 32 (-c32) (-b) 16k (-r 16384)
  53. 53. 2018 iFunFactory Dev Day : 110 / : 27.5 / : 36 ms
  54. 54. 2018 iFunFactory Dev Day ?
  55. 55. 2018 iFunFactory Dev Day #1: CPU
  56. 56. 2018 iFunFactory Dev Day : CPU ? eBPF + BCC: profile • • 30 , 199 flaskr sudo profile -p $(pgrep -nx flaskr)
 -f
 -F 199 30

  57. 57. 2018 iFunFactory Dev Day (1): Flamegraph
  58. 58. 2018 iFunFactory Dev Day (2): DNS C (10%) • __GI_nss_files_servent (OS ) • TCP (6.3%) • (3%) • (2%) ( ) !
  59. 59. 2018 iFunFactory Dev Day CPU ? Linux htop (= top ) • CPU 42% CPU .
  60. 60. 2018 iFunFactory Dev Day #2:
  61. 61. 2018 iFunFactory Dev Day #2 ? HTTP API : • DB • SQL • SQL • etc.
  62. 62. 2018 iFunFactory Dev Day : DB ? eBPF + BCC: mysql_qslower: eBPF + BCC: funclatency • MariaDB (mysql_ )
 sudo funclatency.py -p $(pgrep -nx flaskr)
 -F /usr/lib/mariadb/libmariadb.so:mysql_*
  63. 63. 2018 iFunFactory Dev Day : SQL • : 0.5 ms - 2 ms • 2 ms ≪ 36 ms ( )
  64. 64. 2018 iFunFactory Dev Day : SQL • : 0.008 ms - 0.032 ms • 0.032 ms ≪ 36 ms ( )
  65. 65. 2018 iFunFactory Dev Day
  66. 66. 2018 iFunFactory Dev Day #3: DB
  67. 67. 2018 iFunFactory Dev Day : ? ? • CPU • OS CPU : • OS CPU • CPU =
  68. 68. 2018 iFunFactory Dev Day : eBPF + BCC: offcputime • •
  69. 69. 2018 iFunFactory Dev Day : Flamegraph • poll() : C . 78%; • epoll() : . 16%; boost::asio
  70. 70. 2018 iFunFactory Dev Day : poll() ?
  71. 71. 2018 iFunFactory Dev Day poll() C++ ( ) !
  72. 72. 2018 iFunFactory Dev Day ? • • : • • /
  73. 73. 2018 iFunFactory Dev Day eBPF + BCC: offwaketime : • • offwaketime -p $(pgrep -nx flaskr) -f
  74. 74. 2018 iFunFactory Dev Day • UDP/IP poll
  75. 75. 2018 iFunFactory Dev Day : UDP ?
  76. 76. 2018 iFunFactory Dev Day UDP ? HTTP TCP/IP UDP : • DHCP • DNS • RTP • etc… DNS . ( )
  77. 77. 2018 iFunFactory Dev Day : DNS ?
  78. 78. 2018 iFunFactory Dev Day DNS DNS (C API ) • getnameinfo • getaddrinfo • gethostname • etc… eBPF + BCC: funclatency
  79. 79. 2018 iFunFactory Dev Day • getnameinfo • 32ms - 134ms • (36ms)
  80. 80. 2018 iFunFactory Dev Day : DNS DNS DNS dnsmasq : 1,567 / • 390 / , 2.5 ms : 14 • : 110 / vs. 1,567 /
  81. 81. 2018 iFunFactory Dev Day : MS Windows + ETW GUI + / ETW / Xperf : The Lost Xperf Documentation on CPU Scheduling
  82. 82. 2018 iFunFactory Dev Day 06 
 /
  83. 83. 2018 iFunFactory Dev Day (runqlat) • ( ?) Disk I/O • I/O (fileslower) • (filetop) • ext4slower, btrfsslower,
  84. 84. 2018 iFunFactory Dev Day (memleak) • 30 ? (deadlock_detector) • mutex
  85. 85. 2018 iFunFactory Dev Day Android / https://lwn.net/Articles/742363/
  86. 86. 2018 iFunFactory Dev Day Android / eBPF x86-64 / arm64 1. 
 + LLVM 2. Android 3. 
 4. 
 / / https://github.com/joelagnel/bpfd
  87. 87. 2018 iFunFactory Dev Day Android / BPFd 
 (https://github.com/joelagnel/bpfd) eBPF + BCC (v4.6+; v4.9+) • Android 8.0 Oreo v4.4 AOSP 8.0 v4.10;
  88. 88. 2018 iFunFactory Dev Day 
 
 eBPF + BCC
  89. 89. Q&A 2018 iFunFactory Dev Day 2018 iFunFactory Dev Day Thank You! 660, 1 B 606 +82-70-4923-6566 www.ifunfactory.cominfo@ifunfactory.com

×