This document discusses using strace to trace system calls made by Perl programs. It provides examples of using strace to trace a web application's interactions with a database and memcached. Specifically, it shows strace output for MySQL and memcached requests, and analyzes strace logs to diagnose a slow cache retrieval issue and the "thundering herd problem".

1. strace for Perl
Mongers
zentooo @ DeNA

2. ß…Ãu

3. @zentoo or @zentooo

4. °ntDtÒ–ê±

5. áϱ–©–ÖòÖèÂÇÔ
¨–ß

6. p€g§a†ó

7. àÇ–®w©À{N–ŠgŸ

8. stracet{
Ö캸ÆsPía˜rP—¶¸Åß²–êžtracee—k
zÖê

9. iz¼w|¶¸Åß²–êo
rD”ƒ

10. ¶¸Åß²–ê{|¡Öé°–¶æòX
j[OSz API
File I/O (open, lseek, write, read, close)
Network (socket, bind, listen, accept, connect)
Memory (brk, sbrk, mmap)
Process (fork, wait, kill)
Thread (pthread_create)

11. ¶¸Åß²–êž-,gfPerlsÔ £ê
I/OsY—Wƒ

12. ¶¸Åß²–êUYsPerlscPÖìº
¸žf˜—Wƒ

13. ¶¸Åß²–êUYsPerlsÌÂÇî–®
Öì¯èÞò¯Xca—Wƒ

14. ¡Öé°–¶æò{OSzzzÖ

15. ±¿seX

16. Q. Web¡Öé°–¶æòz›
ÌorD”ƒ

17. A. HTTP鮧¸Çž*]r
HTTPë¸Üò¸ž4e_tl
”„

18. Q. HTTP鮧¸Çž*]r
HTTPë¸Üò¸ž4eoru
RPR_tl”„

19. A. acceptcrreadcrwritecrclosel
”Ϫãì–„

20. ŠNgm®]

21. Web¡ÖézV›Ì
1. HTTP鮧¸Çž*]—
2. _zw‰•zÈèÝXN—
MySQL„z¡®º¸
memcached„z¡®º¸
(Æ–¾z^|logz„Y‹etc)
3. /_zw‰•zÈèÝXN—
4. HTTPë¸Üò¸ž4e

22. ¶¸Åß²–ê§w™e—t...
1. int fd = accept(...); // 鮧¸Ç*]r
2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs
3. _zw‰•zÈèÝXN—
socket, connect, write, read, close
open, write, read, close
4. /_zw‰•zÈèÝXN—
5. write(fd, ...);{{{// ë¸Üò¸4e

23. a•w»ƒe—t
1. int fd = accept(...); // 鮧¸Ç*]r
2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs
3. ‰•zÈèÝ /
4. write(fd, ...);{{{// ë¸Üò¸4e

24. DEMO #1 - just to strace

25. stracež­ŒÔz¬Úò (1)
↓システムコール名 返り値↓
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
↑引数（バッファなどの場合はよしなに展開される）

26. stracež­ŒÔz¬Úò (2)
fd (Ô £êÆ¢¸®éÖ¾) w¶öe—
↓コレ
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308

27. fd (Ô £êÆ¢¸®éÖ¾)
Ô £êÌÂÇî–®¼°ÂÇvuž³ÎBckÖ9
„zÜ£ò¾
(0, 1, 2) = (stdin, stdout, stderr)
i˜’ø{«zw•˜—

28. fd with file
int fd = open(/tmp/yapc2014, ...);
read(fd, buffer);
write(fd, buffer);
close(fd);

29. fd with client socket
int fd = socket();
connect(fd, addr, ...);
read(fd, buffer);
write(fd, buffer);
close(fd);

30. fdžðRzXè2z2˜
Ô £êzf#
opens4orYkfdXclosewUa˜—Šs
®è£¡òÇ´£Èsz¼°ÂÇzf#
sockets4orYkfdXclosewUa˜—Šs
´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf#
accepts4orYkfdXclosewUa˜—Šs

31. Web¡Öézstracež­ŒÔz¬Úò
// こっから
accept(4, {sa_family=AF_INET, ...)}, [16]) = 5
(中略)
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
(中略)
write(5, HTTP/1.1 200 OKrnDate: Sun, 10 A..., 2218) = 2218
// ここまでが一つのリクエスト/レスポンス
// keepaliveしてないならここで close

32. DEMO #2 - strace web app

33. N[Šs2˜vzsº“Ž
Twiggy, Twiggy::Preforkvuevent-drivenv´–Ï
(epoll_wait)
multi socket listencr—ÔzStarlet (select)

34. Web¡ÖézV›Ì($)
lPkPzÝ „Ðzó
sþUwv—
HTTP鮧¸Çž*]—
_zw‰•zÈèÝXN—
MySQL„z¡®º¸
memcached„z¡®º¸
(Æ–¾z^|logz„Y‹etc)
/_zw‰•zÈèÝXN—
HTTPë¸Üò¸ž4e

35. ¶öÜ£òÇ - MySQL
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10
fcntl(10, F_SETFL, O_RDONLY) = 0
fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(10, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr(10.0.0.1
fcntl(10, F_SETFL, O_RDWR) = 0
poll([{fd=10, events=POLLIN|POLLPRI}], 1, 4000) = 1 ([{fd=10, revents=POLLIN}
setsockopt(10, SOL_SOCKET, SO_RCVTIMEO, 200334110000000setsockopt(10, SOL_SOCKET, SO_SNDTIMEO, 200334110000000setsockopt(10, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

36. ¶öÜ£òÇ - memcached
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
fcntl(8, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(8, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr(10.0.0.1
poll([{fd=8, events=POLLOUT}], 1, 250) = 1 ([{fd=8, revents=POLLOUT}])
getsockopt(8, SOL_SOCKET, SO_ERROR, [-511598409602301952], [4]) = 0
setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendmsg(8, {msg_name(0)=NULL, msg_iov(6)=[{set, 3}, { csrf_token:, 20},
read(8, 0x59f9f50, 1536) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=8, events=POLLIN}], 1, 300) = 1 ([{fd=8, revents=POLLIN}])
read(8, STOREDrn, 1536)

37. DEMO #3 - strace web app
with DB and memcached

38. Öì¯èß{­oktV–w
”WvP
•d®§éžDxPcr—
­ork”–cache misscr—
Ï7wretrycr—
TCP ConnectionžPŠ›grvP
TCP ConnectionmŸtˆ˜rvP

39. öº #1 - memcached retry
problem

40. +180ms
tN—APIzë¸Üò¸X|éé–¸ w180msËB
50ms or diev•4x[•P¶vvPtP]vPëØê

41. ?_
Cache::Memcached::FastzèÂЖá·ä–ê
(Ò1
XNokzs)

42. M[èo
strace -p $pid -e 'trace=sendmsg'
(Cache::Memcached::FastXsendmsgžžRz{—orPk)

43. Ë
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13}
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13}
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13}
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13}
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,

44. ̾|•c
èÂЖá·ä–êwretryAÿ (maxs3x) XâorPk
Cache::Memcached::Fastzget{cache misstiz)z§
è–žæ*sYvP
cache missz_w3xretrycrk
retry¹{20msec
20 msec × 3 (retry) × 3 = 180 msec
izhit rate{uRvŸƒtPRø{N–ppŽget,{
retrycvP7cžâ˜rwE

45. Ë - with timestamp
22:45:34.517703 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.537967 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.558936 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.579275 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.587428 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:34.608296 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:34.628673 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:34.649146 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:35.135129 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:35.155516 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:35.176027 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
22:45:35.196453 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},

46. stracecr‹—t´ö¿
ÙžpWvPÖêwc—

47. ÙžpWvPÖê
strace
tcpdump
gdb

48. Ùžp[Žz
zM6
¡Öézì¯
Ôw{¡Öéz²–È

49. öº #2 - accept(2)
thundering herd

50. What is thundering herd
The thundering herd problem occurs when a large
number of processes waiting for an event are awoken
when that event occurs, but only one process is able to
proceed at a time. - Wikipedia
*[zÖ캸X§zÇé«–s•Ôw”YceX|ö
Uz{v{´zÖ캸scW훘vP (Šk{í›
˜—…Y) Ôwé¼–¸XÏ7wv—þU
”[N—z{Æ–¾zremote cacheXexpireckÔw´D
wDB„z®§éX—|tW

51. accept(2) and prefork server
DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j
˜Õì®
Linux kernelX”cvwqueueingcr[˜—zs|鮧
¸ÇXakÔwOYÖX—Ö캸{´pl]
(ˆæ)
http://d.hatena.ne.jp/naoya/20070311/1173629378

52. accept(2) and event-driven/
multi-process server
DÖ캸X•Ôw´pzsocket fdžepoll_waitsÁ
listen socketXreadIÿ = acceptXIÿwvokÔFs
epoll_waitcr—ÀrzÖ캸XOYÖX—
´pzÖ캸Xacceptw´
)zÖ캸{óorî (EAGAIN)

53. accept(2) thundering herd
accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s
Pf
loadXÖX—
vmstatsf—t...
ustsytcsz–Xg–ÖX— (50, 50, 100000tW)
Ö캸zcontext switch
TeZXOÀWt­›˜—

54. DEMO #3 - EAGAIN and again
and again

55. wEn
iŽiŽXevent-drivenvŸlciŸvwR£wÖ캸
N^v[rPP”y

56. vv,w”–100Ö캸[•Pw (¼{300[•PN
ok)

57. 7•
xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr
P—
Monoceros‹kPwaccepte—Ö캸t鮧¸Çž
{ve—Ö캸žñÀw
]—tWƒ
event-driventÝêÀÖ캸{øPgXuP with Perl
IO::AIO (AnyEvent::IOzbackend) tWŽÇt¶ò¯êÖ
캸¼völok–...

58. Štwâ–Še

59. stracez›—Pt_™
ß¿váQXVVP (ø˜˜|uRwsŽv—X)
kŠwO_—,zÌÎw{¹PrvP (Devel::KYTProftW
sì¯cck[X”Pƒ¥

60. stracez”Pt_™
̼XÓPs|Ö캸wattache—l]sef˜
—
Web¡Öé°–¶æòzDÓmX
W—

61. stracelt
W•vP_t
¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v
¡Öé°–¶æòszÏ7loop, deep recursion
µzuP¡ê³é¹ßtW
-d NYTProf
Çèò¸Ü–Çä’AszþU
tcpdumptWtsharktWngreptWžPŠc“R

62. straces
W—_t
鮧¸Ç*]r|ë¸Üò¸4eŠszÔì–
Æ–¾¡®º¸zöA
öUwDxmemcachedW•getcr—W|DxDB‹w
Por—W
öUwcachea˜rP—data
öUwPícrP—SELECT
iz)|ŽP z6crPvP“”

63. H
|k–¼l]ustrace{¸ÿzÖêdvP
žT|ÚàW•ä—tWiŸv_tŽvP
sŽ|žP[X
WorP˜|V7€­¦
Web¡ÖézDÓmX
W—

64. ˆæ`
The Linux Programing Interface
ŠPc|K¨„€h(÷§cWvP]u...)Vee
à˜§hŽN—]u1000Ù–·Ý|pŠ–„€s{v[
ï1
À󭌫Pvc|éÔ ëò¸tcrŽ6q
äz˜{Veel]u|±¿­Œz{ޏXûPzs
pRzLinuxÖì¯èÞò¯
tWz[X?|{PPWŽc
˜vP

65. ™G (?)
Web+DBz2014ñ10KÿwMÌ(Perl Hackers Hub)ž„Wg
rá[_twv–Šck

66. `s×N–XtR`bPŠc
k