strace for Perl 
Mongers 
zentooo @ DeNA
߅Ãu
@zentoo or @zentooo
°ntDtҖê±
áϱ–©–ÖòÖèÂÇÔ 
¨–ß
p€g§a†ó
àǖ®w©À{N–ŠgŸ
stracet{ 
Ö캸ÆsPía˜rP—¶¸Åß²–êžtracee—k 
zÖê
iz¼w|¶¸Åß²–êo 
rD”ƒ
¶¸Åß²–ê{|¡Ö鰖¶æòX 
j[OSz API 
File I/O (open, lseek, write, read, close) 
Network (socket, bind, listen, accept, connect)...
¶¸Åß²–êž-,gfPerlsÔ £ê 
I/OsY—Wƒ
¶¸Åß²–êUYsPerlscPÖìº 
¸žf˜—Wƒ
¶¸Åß²–êUYsPerlsÌÂÇ 
Öì¯èÞò¯Xca—Wƒ
¡Ö鰖¶æò{OSzzzÖ
±¿seX
Q. Web¡Ö鰖¶æòz› 
ÌorD”ƒ
A. HTTP鮧¸Çž*]r 
HTTPë¸Üò¸ž4e_tl 
”„
Q. HTTP鮧¸Çž*]r 
HTTPë¸Üò¸ž4eoru 
RPR_tl”„
A. acceptcrreadcrwritecrclosel 
”Ϫã얄
ŠNgm®]
Web¡ÖézV›Ì 
1. HTTP鮧¸Çž*]— 
2. _zw‰•zÈèÝXN— 
MySQL„z¡®º¸ 
memcached„z¡®º¸ 
(Ɩ¾z^|logz„Y‹etc) 
3. /_zw‰•zÈèÝXN— 
4. HTTP...
¶¸Åß²–ê§w™e—t... 
1. int fd = accept(...); // 鮧¸Ç*]r 
2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 
3. _zw‰•zÈèÝXN— 
socket, conne...
a•w»ƒe—t 
1. int fd = accept(...); // 鮧¸Ç*]r 
2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 
3. ‰•zÈèÝ / 
4. write(fd, ...);{{{// ë...
DEMO #1 - just to strace
stracež­ŒÔz¬Úò (1) 
↓システムコール名 返り値↓ 
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 
↑引数(バッファなどの場合はよしなに展開される)
stracež­ŒÔz¬Úò (2) 
fd (Ô £êÆ¢¸®éÖ¾) w¶öe— 
↓コレ 
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
fd (Ô £êÆ¢¸®éÖ¾) 
Ô £êÌÂÇ¼°ÂÇvuž³ÎBckÖ9 
„zÜ£ò¾ 
(0, 1, 2) = (stdin, stdout, stderr) 
i˜’ø{«zw•˜—
fd with file 
int fd = open(/tmp/yapc2014, ...); 
read(fd, buffer); 
write(fd, buffer); 
close(fd);
fd with client socket 
int fd = socket(); 
connect(fd, addr, ...); 
read(fd, buffer); 
write(fd, buffer); 
close(fd);
fdžðRzXè2z2˜ 
Ô £êzf# 
opens4orYkfdXclosewUa˜—Šs 
®è£¡òÇ´£Èsz¼°ÂÇzf# 
sockets4orYkfdXclosewUa˜—Šs 
´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf#...
Web¡Öézstracež­ŒÔz¬Úò 
// こっから 
accept(4, {sa_family=AF_INET, ...)}, [16]) = 5 
(中略) 
read(5, GET / HTTP/1.0rnHost: xxx-xx...
DEMO #2 - strace web app
N[Šs2˜vzsº“Ž 
Twiggy, Twiggy::Preforkvuevent-drivenv´–Ï 
(epoll_wait) 
multi socket listencr—ÔzStarlet (select)
Web¡ÖézV›Ì($) 
lPkPzÝ „Ðzó
 sþUwv— 
HTTP鮧¸Çž*]— 
_zw‰•zÈèÝXN— 
MySQL„z¡®º¸ 
memcached„z¡®º¸ 
(Ɩ¾z^|logz„Y‹etc) 
/_zw‰•z...
¶öÜ£òÇ - MySQL 
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10 
fcntl(10, F_SETFL, O_RDONLY) = 0 
fcntl(10, F_GETFL) = 0x2 ...
¶öÜ£òÇ - memcached 
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 
fcntl(8, F_GETFL) = 0x2 (flags O_RDWR) 
fcntl(8, F_SETF...
DEMO #3 - strace web app 
with DB and memcached
Öì¯èß{­oktV–w 
”WvP 
•d®§éžDxPcr— 
­ork”–cache misscr— 
Ï7wretrycr— 
TCP ConnectionžPŠ›grvP 
TCP ConnectionmŸtˆ˜rvP
öº #1 - memcached retry 
problem
+180ms 
tN—APIzë¸Üò¸X|é閸 w180msËB 
50ms or diev•4x[•P¶vvPtP]vPëØê
?_	 
Cache::Memcached::FastzèÂЖá·ä–ê 
(Ò1
XNokzs)
M[èo 
strace -p $pid -e 'trace=sendmsg' 
(Cache::Memcached::FastXsendmsgžžRz{—orPk)
Ë 
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} 
sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get...
̾|•c 
èÂЖá·ä–êwretryAÿ (maxs3x) XâorPk 
Cache::Memcached::Fastzget{cache misstiz)z§ 
薞æ*sYvP 
cache missz_w3xretrycrk ...
Ë - with timestamp 
22:45:34.517703 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.537967 sendmsg(6, {msg_...
stracecr‹—t´ö¿ 
ٞpWvPÖêwc—
ٞpWvPÖê 
strace 
tcpdump 
gdb
ٞp[Žz 
zM6 
¡Öézì¯ 
Ôw{¡Öéz²–È
öº #2 - accept(2) 
thundering herd
What is thundering herd 
The thundering herd problem occurs when a large 
number of processes waiting for an event are awo...
accept(2) and prefork server 
DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j 
˜Õì® 
Linux kernelX”cvwqueueingcr[˜—zs|鮧 
¸ÇXakÔwOYÖX—...
accept(2) and event-driven/ 
multi-process server 
DÖ캸X•Ôw´pzsocket fdžepoll_waitsÁ 
listen socketXreadIÿ = acceptXIÿwvo...
accept(2) thundering herd 
accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s 
Pf 
loadXÖX— 
vmstatsf—t... 
ustsytcsz–Xg–ÖX— (50, 50, 100000tW...
DEMO #3 - EAGAIN and again 
and again
wEn 
iŽiŽXevent-drivenvŸlciŸvwR£wÖ캸 
N^v[rPP”y
vv,w”–100Ö캸[•Pw (¼{300[•PN 
ok)
7• 
xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr 
P— 
Monoceros‹kPwaccepte—Ö캸t鮧¸Çž 
{ve—Ö캸žñÀw
]—tWƒ 
event-driventÝêÀÖ캸{øPgXuP with...
Štw▊e
stracez›—Pt_™ 
ß¿váQXVVP (ø˜˜|uRwsŽv—X) 
kŠwO_—,zÌÎw{¹PrvP (Devel::KYTProftW 
sì¯cck[X”Pƒ¥
stracez”Pt_™ 
̼XÓPs|Ö캸wattache—l]sef˜ 
— 
Web¡Ö鰖¶æòzDÓmX
W—
stracelt
W•vP_t 
¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v 
¡Ö鰖¶æòszÏ7loop, deep recursion 
µzuP¡ê³é¹ßtW 
-d NYTProf 
Çèò¸Ü–Çä’AszþU 
...
straces
W—_t 
鮧¸Ç*]r|ë¸Üò¸4eŠszÔì– 
Ɩ¾¡®º¸zöA 
öUwDxmemcachedW•getcr—W|DxDB‹w 
Por—W 
öUwcachea˜rP—data 
öUwPícrP—SELEC...
H 
|k–¼l]ustrace{¸ÿzÖêdvP 
žT|ÚàW•ä—tWiŸv_tŽvP 
sŽ|žP[X
WorP˜|V7€­¦ 
Web¡ÖézDÓmX
W—
ˆæ` 
The Linux Programing Interface 
ŠPc|K¨„€h(÷§cWvP]u...)Vee 
à˜§hŽN—]u1000ٖ·Ý|pŠ–„€s{v[ 
ï1 
À󭌫Pvc|éÔ ëò¸tcrŽ6q 
ä...
™G (?) 
Web+DBz2014ñ10KÿwMÌ(Perl Hackers Hub)ž„Wg 
rá[_twv–Šck
Upcoming SlideShare
Loading in …5
×

strace for Perl Mongers

1,949 views

Published on

YAPC::Asia 2014の発表スライドです。

Published in: Technology
  • Be the first to comment

strace for Perl Mongers

  1. 1. strace for Perl Mongers zentooo @ DeNA
  2. 2. ߅Ãu
  3. 3. @zentoo or @zentooo
  4. 4. °ntDtҖê±
  5. 5. áϱ–©–ÖòÖèÂÇÔ ¨–ß
  6. 6. p€g§a†ó
  7. 7. àǖ®w©À{N–ŠgŸ
  8. 8. stracet{ Ö캸ÆsPía˜rP—¶¸Åß²–êžtracee—k zÖê
  9. 9. iz¼w|¶¸Åß²–êo rD”ƒ
  10. 10. ¶¸Åß²–ê{|¡Ö鰖¶æòX j[OSz API File I/O (open, lseek, write, read, close) Network (socket, bind, listen, accept, connect) Memory (brk, sbrk, mmap) Process (fork, wait, kill) Thread (pthread_create)
  11. 11. ¶¸Åß²–êž-,gfPerlsÔ £ê I/OsY—Wƒ
  12. 12. ¶¸Åß²–êUYsPerlscPÖìº ¸žf˜—Wƒ
  13. 13. ¶¸Åß²–êUYsPerlsÌÂÇ Öì¯èÞò¯Xca—Wƒ
  14. 14. ¡Ö鰖¶æò{OSzzzÖ
  15. 15. ±¿seX
  16. 16. Q. Web¡Ö鰖¶æòz› ÌorD”ƒ
  17. 17. A. HTTP鮧¸Çž*]r HTTPë¸Üò¸ž4e_tl ”„
  18. 18. Q. HTTP鮧¸Çž*]r HTTPë¸Üò¸ž4eoru RPR_tl”„
  19. 19. A. acceptcrreadcrwritecrclosel ”Ϫã얄
  20. 20. ŠNgm®]
  21. 21. Web¡ÖézV›Ì 1. HTTP鮧¸Çž*]— 2. _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Ɩ¾z^|logz„Y‹etc) 3. /_zw‰•zÈèÝXN— 4. HTTPë¸Üò¸ž4e
  22. 22. ¶¸Åß²–ê§w™e—t... 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 3. _zw‰•zÈèÝXN— socket, connect, write, read, close open, write, read, close 4. /_zw‰•zÈèÝXN— 5. write(fd, ...);{{{// ë¸Üò¸4e
  23. 23. a•w»ƒe—t 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 3. ‰•zÈèÝ / 4. write(fd, ...);{{{// ë¸Üò¸4e
  24. 24. DEMO #1 - just to strace
  25. 25. stracež­ŒÔz¬Úò (1) ↓システムコール名 返り値↓ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 ↑引数(バッファなどの場合はよしなに展開される)
  26. 26. stracež­ŒÔz¬Úò (2) fd (Ô £êÆ¢¸®éÖ¾) w¶öe— ↓コレ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
  27. 27. fd (Ô £êÆ¢¸®éÖ¾) Ô £êÌÂÇ¼°ÂÇvuž³ÎBckÖ9 „zÜ£ò¾ (0, 1, 2) = (stdin, stdout, stderr) i˜’ø{«zw•˜—
  28. 28. fd with file int fd = open(/tmp/yapc2014, ...); read(fd, buffer); write(fd, buffer); close(fd);
  29. 29. fd with client socket int fd = socket(); connect(fd, addr, ...); read(fd, buffer); write(fd, buffer); close(fd);
  30. 30. fdžðRzXè2z2˜ Ô £êzf# opens4orYkfdXclosewUa˜—Šs ®è£¡òÇ´£Èsz¼°ÂÇzf# sockets4orYkfdXclosewUa˜—Šs ´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf# accepts4orYkfdXclosewUa˜—Šs
  31. 31. Web¡Öézstracež­ŒÔz¬Úò // こっから accept(4, {sa_family=AF_INET, ...)}, [16]) = 5 (中略) read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 (中略) write(5, HTTP/1.1 200 OKrnDate: Sun, 10 A..., 2218) = 2218 // ここまでが一つのリクエスト/レスポンス // keepaliveしてないならここで close
  32. 32. DEMO #2 - strace web app
  33. 33. N[Šs2˜vzsº“Ž Twiggy, Twiggy::Preforkvuevent-drivenv´–Ï (epoll_wait) multi socket listencr—ÔzStarlet (select)
  34. 34. Web¡ÖézV›Ì($) lPkPzÝ „Ðzó sþUwv— HTTP鮧¸Çž*]— _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Ɩ¾z^|logz„Y‹etc) /_zw‰•zÈèÝXN— HTTPë¸Üò¸ž4e
  35. 35. ¶öÜ£òÇ - MySQL socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10 fcntl(10, F_SETFL, O_RDONLY) = 0 fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(10, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr(10.0.0.1 fcntl(10, F_SETFL, O_RDWR) = 0 poll([{fd=10, events=POLLIN|POLLPRI}], 1, 4000) = 1 ([{fd=10, revents=POLLIN} setsockopt(10, SOL_SOCKET, SO_RCVTIMEO, 200334110000000setsockopt(10, SOL_SOCKET, SO_SNDTIMEO, 200334110000000setsockopt(10, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0 setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
  36. 36. ¶öÜ£òÇ - memcached socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 fcntl(8, F_GETFL) = 0x2 (flags O_RDWR) fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(8, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr(10.0.0.1 poll([{fd=8, events=POLLOUT}], 1, 250) = 1 ([{fd=8, revents=POLLOUT}]) getsockopt(8, SOL_SOCKET, SO_ERROR, [-511598409602301952], [4]) = 0 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 sendmsg(8, {msg_name(0)=NULL, msg_iov(6)=[{set, 3}, { csrf_token:, 20}, read(8, 0x59f9f50, 1536) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=8, events=POLLIN}], 1, 300) = 1 ([{fd=8, revents=POLLIN}]) read(8, STOREDrn, 1536)
  37. 37. DEMO #3 - strace web app with DB and memcached
  38. 38. Öì¯èß{­oktV–w ”WvP •d®§éžDxPcr— ­ork”–cache misscr— Ï7wretrycr— TCP ConnectionžPŠ›grvP TCP ConnectionmŸtˆ˜rvP
  39. 39. öº #1 - memcached retry problem
  40. 40. +180ms tN—APIzë¸Üò¸X|é閸 w180msËB 50ms or diev•4x[•P¶vvPtP]vPëØê
  41. 41. ?_ Cache::Memcached::FastzèÂЖá·ä–ê (Ò1 XNokzs)
  42. 42. M[èo strace -p $pid -e 'trace=sendmsg' (Cache::Memcached::FastXsendmsgžžRz{—orPk)
  43. 43. Ë sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
  44. 44. ̾|•c èÂЖá·ä–êwretryAÿ (maxs3x) XâorPk Cache::Memcached::Fastzget{cache misstiz)z§ 薞æ*sYvP cache missz_w3xretrycrk retry¹{20msec 20 msec × 3 (retry) × 3 = 180 msec izhit rate{uRvŸƒtPRø{N–ppŽget,{ retrycvP7cžâ˜rwE
  45. 45. Ë - with timestamp 22:45:34.517703 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.537967 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.558936 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.579275 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.587428 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.608296 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.628673 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.649146 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.135129 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.155516 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.176027 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.196453 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
  46. 46. stracecr‹—t´ö¿ ٞpWvPÖêwc—
  47. 47. ٞpWvPÖê strace tcpdump gdb
  48. 48. ٞp[Žz zM6 ¡Öézì¯ Ôw{¡Öéz²–È
  49. 49. öº #2 - accept(2) thundering herd
  50. 50. What is thundering herd The thundering herd problem occurs when a large number of processes waiting for an event are awoken when that event occurs, but only one process is able to proceed at a time. - Wikipedia *[zÖ캸X§zÇ髖s•Ôw”YceX|ö Uz{v{´zÖ캸scW훘vP (Šk{í› ˜—…Y) Ôw鼖¸XÏ7wv—þU ”[N—z{Ɩ¾zremote cacheXexpireckÔw´D wDB„z®§éX—|tW
  51. 51. accept(2) and prefork server DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j ˜Õì® Linux kernelX”cvwqueueingcr[˜—zs|鮧 ¸ÇXakÔwOYÖX—Ö캸{´pl] (ˆæ) http://d.hatena.ne.jp/naoya/20070311/1173629378
  52. 52. accept(2) and event-driven/ multi-process server DÖ캸X•Ôw´pzsocket fdžepoll_waitsÁ listen socketXreadIÿ = acceptXIÿwvokÔFs epoll_waitcr—ÀrzÖ캸XOYÖX— ´pzÖ캸Xacceptw´ )zÖ캸{óorî (EAGAIN)
  53. 53. accept(2) thundering herd accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s Pf loadXÖX— vmstatsf—t... ustsytcsz–Xg–ÖX— (50, 50, 100000tW) Ö캸zcontext switch TeZXOÀWt­›˜—
  54. 54. DEMO #3 - EAGAIN and again and again
  55. 55. wEn iŽiŽXevent-drivenvŸlciŸvwR£wÖ캸 N^v[rPP”y
  56. 56. vv,w”–100Ö캸[•Pw (¼{300[•PN ok)
  57. 57. 7• xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr P— Monoceros‹kPwaccepte—Ö캸t鮧¸Çž {ve—Ö캸žñÀw ]—tWƒ event-driventÝêÀÖ캸{øPgXuP with Perl IO::AIO (AnyEvent::IOzbackend) tWŽÇt¶ò¯êÖ ìº¸¼völok–...
  58. 58. Štw▊e
  59. 59. stracez›—Pt_™ ß¿váQXVVP (ø˜˜|uRwsŽv—X) kŠwO_—,zÌÎw{¹PrvP (Devel::KYTProftW sì¯cck[X”Pƒ¥
  60. 60. stracez”Pt_™ ̼XÓPs|Ö캸wattache—l]sef˜ — Web¡Ö鰖¶æòzDÓmX W—
  61. 61. stracelt W•vP_t ¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v ¡Ö鰖¶æòszÏ7loop, deep recursion µzuP¡ê³é¹ßtW -d NYTProf Çèò¸Ü–Çä’AszþU tcpdumptWtsharktWngreptWžPŠc“R
  62. 62. straces W—_t 鮧¸Ç*]r|ë¸Üò¸4eŠszÔì– Æ–¾¡®º¸zöA öUwDxmemcachedW•getcr—W|DxDB‹w Por—W öUwcachea˜rP—data öUwPícrP—SELECT iz)|ŽP z6crPvP“”
  63. 63. H |k–¼l]ustrace{¸ÿzÖêdvP žT|ÚàW•ä—tWiŸv_tŽvP sŽ|žP[X WorP˜|V7€­¦ Web¡ÖézDÓmX W—
  64. 64. ˆæ` The Linux Programing Interface ŠPc|K¨„€h(÷§cWvP]u...)Vee à˜§hŽN—]u1000ٖ·Ý|pŠ–„€s{v[ ï1 À󭌫Pvc|éÔ ëò¸tcrŽ6q äz˜{Veel]u|±¿­Œz{ޏXûPzs pRzLinuxÖì¯èÞò¯ tWz[X?|{PPWŽc ˜vP
  65. 65. ™G (?) Web+DBz2014ñ10KÿwMÌ(Perl Hackers Hub)ž„Wg rá[_twv–Šck
  66. 66. `s×N–XtR`bPŠc k

×