[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

2018 iFunFactory Dev Day
(2013-)
• (C++, C#)
• (Python)
(2012-2013)
• (Python)
(2007-2012)
• (C++)
• (C++, C#, Python)
• (C#)

Contents.
01
02
03
04
05
06

?
?
Linux eBPF + BCC

:
• (2017 )
• +
•
• DB /
• CPU
• (lock)
• (DB, redis, 3rd party API)
•

:
• / DB / CPU ( / / / )
• DB
• /

•
.
• /
• / / (fiber) /
•
• /
•
•

• / .
• .
•
•
• / /

• /

: CPU
• GNU Compiler Collection (GCC)
• -pg (gprof)
• LLVM/Clang
• -fprofile-instr-generate
• Visual C++ instrumentation
• Valgrind Callgrind

• /
• /
•
• /

GProf :
% cumulative self self total
time seconds seconds calls ms/call ms/call name
14.29 0.03 0.03 207323 0.00 0.00 crow::json::escape
14.29 0.06 0.03 3059 0.01 0.02 crow::CrowApp::handle
9.52 0.08 0.02 2638 0.01 0.02 crow::json::dump_internal
4.76 0.09 0.01 307190 0.00 0.00 _Hashtable<…>::insert_unique
4.76 0.10 0.01 208356 0.00 0.00 _Hashtable<…>::_M_rehash
4.76 0.11 0.01 19129 0.00 0.00
vector<crow::json::wvalue>::_M_emplace_back_aux
4.76 0.12 0.01 11961 0.00 0.00
crow::Connection<crow::SocketAdaptor, crow::CrowApp>::cancel_deadline_timer

GProf :

• gprof: 30% - 300%
• Valgrind: 400% - 4,000%
• 3rd (.dll, .so)
•

.
/
• (CPU, , , )
•
•

• IO 10ms ?
• ?
• ?

CPU
• .
• Linux perf (kernel v2.6+)
• Linux eBPF (kernel v3.18+; v4.9+)
• Google Performance Tools
• Visual C++ : CPU

• ( 50 - 200 )
• CPU
•
• /

: CPU
Linux v4.13 eBPF + BCC profile:
Flamegraph: /  
https://github.com/brendangregg/FlameGraph

 
OS
/
( ), OS

OS
OS
OS !
• Linux v4.13 randstruct
• eBPF #define

?
•Disk I/O ?
•CPU ?
• (lock) ?
• API ?
• ?
• ?

Linux
•eBPF (kernel v3.18+; 4.9+ )
•perf (kernel v2.6+; )
Windows
•Event Tracing for Windows (ETW; Win 8+)
•Windows Event Logging ( ; Win 2k+)

?
/ ?

Linux eBPF + BCC
• eBPF: /
• BCC: Python / lua / go eBPF

eBPF
OS  
(x86-64/arm64 )
VM
• / ( )
•
• JIT

eBPF: ?
•
• (= )
• ( )
•
( )

BCC
• eBFP Compiler Collection
• Python / lua / go /
• C eBPF
• eBPF
• python / lua / go

eBPF + BCC:
CPU, , I/O,
eBPF + BCC
( )
:
1. ,
2. eBPF + BCC
3.

BCC :
BCC funclatency
/

: eBPF (1)

: eBPF (2)

: BCC + BPF

Linux Kernel v4.13+
randstruct .
BCC “bpf_text”
#define randomized_struct_fields_start struct {
#define randomized_struct_fields_end };

05 HTTP API

HTTP API
I/O, DB , ( )
/OS
: flaskr
• Crow HTTP
• CrowDB + MariaDB
• RESTful API
• https://github.com/jinukkim/devday-2018-03

HTTP :
URL / HTTP
HTTP , 
4

HTTP :
DB
JSON
HTTP

HTTP :
MariaDB DB
HTTP DNS

4 1000
250
≦ 4 ms

siege: HTTP
siege -c 32 
-r 16384 
-b 
http://example.com
URL 32 (-c32)
(-b)
16k (-r 16384)

:
110 /
: 27.5 /
: 36 ms

#1: CPU

: CPU ?
eBPF + BCC: profile
•
•
30 , 199 flaskr
sudo profile -p $(pgrep -nx flaskr) 
-f 
-F 199 30

(1): Flamegraph

(2):
DNS C (10%)
• __GI_nss_files_servent
(OS )
• TCP (6.3%)
• (3%)
• (2%)
( ) !

CPU ?
Linux htop (= top )
• CPU 42% CPU .

#2 ?
HTTP API
:
• DB
• SQL
• SQL
• etc.

: DB ?
eBPF + BCC: mysql_qslower:
eBPF + BCC: funclatency
•
MariaDB (mysql_ ) 
sudo funclatency.py -p $(pgrep -nx flaskr) 
-F /usr/lib/mariadb/libmariadb.so:mysql_*

: SQL
• : 0.5 ms - 2 ms
• 2 ms ≪ 36 ms ( )

: SQL
• : 0.008 ms - 0.032 ms
• 0.032 ms ≪ 36 ms ( )

#3: DB

:
?
?
• CPU
• OS CPU
:
• OS CPU
• CPU =

:
eBPF + BCC: offcputime
•
•

: Flamegraph
• poll() : C . 78%;
• epoll() : . 16%; boost::asio

: poll() ?

poll()
C++
( )
!

?
•
• :
•
• /

eBPF + BCC: offwaketime
:
•
•
offwaketime -p $(pgrep -nx flaskr) -f

• UDP/IP poll

: UDP ?

UDP ?
HTTP TCP/IP
UDP :
• DHCP
• DNS
• RTP
• etc…
DNS . ( )

: DNS ?

DNS
DNS (C API )
• getnameinfo
• getaddrinfo
• gethostname
• etc…
eBPF + BCC: funclatency

• getnameinfo
• 32ms - 134ms
• (36ms)

: DNS
DNS DNS dnsmasq
: 1,567 /
• 390 / , 2.5 ms
: 14
• : 110 / vs. 1,567 /

: MS Windows + ETW
GUI + /
ETW / Xperf
: The Lost Xperf Documentation on CPU Scheduling

06  
/

(runqlat)
• ( ?)
Disk I/O
• I/O (fileslower)
• (filetop)
• ext4slower, btrfsslower,

(memleak)
• 30 ?
(deadlock_detector)
• mutex

Android /
https://lwn.net/Articles/742363/

Android /
eBPF
x86-64 / arm64
1.  
+ LLVM
2. Android
3.  
4.  
/ /
https://github.com/joelagnel/bpfd

Android /
BPFd  
(https://github.com/joelagnel/bpfd)
eBPF + BCC (v4.6+; v4.9+)
• Android 8.0 Oreo v4.4
AOSP 8.0 v4.10;

   
eBPF + BCC

Q&A
Thank You!
660, 1 B 606
+82-70-4923-6566 www.ifunfactory.cominfo@ifunfactory.com

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

More Related Content

What's hot

Similar to [아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

More from iFunFactory Inc.

Recently uploaded

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석