Your SlideShare is downloading. ×
strace for Perl Mongers
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

strace for Perl Mongers

1,073
views

Published on

YAPC::Asia 2014の発表スライドです。

YAPC::Asia 2014の発表スライドです。

Published in: Technology

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,073
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
1
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. strace for Perl Mongers zentooo @ DeNA
  • 2. ß…Ãu
  • 3. @zentoo or @zentooo
  • 4. °ntDtÒ–ê±
  • 5. áϱ–©–ÖòÖèÂÇÔ ¨–ß
  • 6. p€g§a†ó
  • 7. àÇ–®w©À{N–ŠgŸ
  • 8. stracet{ Ö캸ÆsPía˜rP—¶¸Åß²–êžtracee—k zÖê
  • 9. iz¼w|¶¸Åß²–êo rD”ƒ
  • 10. ¶¸Åß²–ê{|¡Öé°–¶æòX j[OSz API File I/O (open, lseek, write, read, close) Network (socket, bind, listen, accept, connect) Memory (brk, sbrk, mmap) Process (fork, wait, kill) Thread (pthread_create)
  • 11. ¶¸Åß²–êž-,gfPerlsÔ £ê I/OsY—Wƒ
  • 12. ¶¸Åß²–êUYsPerlscPÖìº ¸žf˜—Wƒ
  • 13. ¶¸Åß²–êUYsPerlsÌÂÇî–® Öì¯èÞò¯Xca—Wƒ
  • 14. ¡Öé°–¶æò{OSzzzÖ
  • 15. ±¿seX
  • 16. Q. Web¡Öé°–¶æòz› ÌorD”ƒ
  • 17. A. HTTP鮧¸Çž*]r HTTPë¸Üò¸ž4e_tl ”„
  • 18. Q. HTTP鮧¸Çž*]r HTTPë¸Üò¸ž4eoru RPR_tl”„
  • 19. A. acceptcrreadcrwritecrclosel ”Ϫãì–„
  • 20. ŠNgm®]
  • 21. Web¡ÖézV›Ì 1. HTTP鮧¸Çž*]— 2. _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Æ–¾z^|logz„Y‹etc) 3. /_zw‰•zÈèÝXN— 4. HTTPë¸Üò¸ž4e
  • 22. ¶¸Åß²–ê§w™e—t... 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 3. _zw‰•zÈèÝXN— socket, connect, write, read, close open, write, read, close 4. /_zw‰•zÈèÝXN— 5. write(fd, ...);{{{// ë¸Üò¸4e
  • 23. a•w»ƒe—t 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?­Ÿs 3. ‰•zÈèÝ / 4. write(fd, ...);{{{// ë¸Üò¸4e
  • 24. DEMO #1 - just to strace
  • 25. stracež­ŒÔz¬Úò (1) ↓システムコール名 返り値↓ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 ↑引数(バッファなどの場合はよしなに展開される)
  • 26. stracež­ŒÔz¬Úò (2) fd (Ô £êÆ¢¸®éÖ¾) w¶öe— ↓コレ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
  • 27. fd (Ô £êÆ¢¸®éÖ¾) Ô £êÌÂÇî–®¼°ÂÇvuž³ÎBckÖ9 „zÜ£ò¾ (0, 1, 2) = (stdin, stdout, stderr) i˜’ø{«zw•˜—
  • 28. fd with file int fd = open(/tmp/yapc2014, ...); read(fd, buffer); write(fd, buffer); close(fd);
  • 29. fd with client socket int fd = socket(); connect(fd, addr, ...); read(fd, buffer); write(fd, buffer); close(fd);
  • 30. fdžðRzXè2z2˜ Ô £êzf# opens4orYkfdXclosewUa˜—Šs ®è£¡òÇ´£Èsz¼°ÂÇzf# sockets4orYkfdXclosewUa˜—Šs ´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf# accepts4orYkfdXclosewUa˜—Šs
  • 31. Web¡Öézstracež­ŒÔz¬Úò // こっから accept(4, {sa_family=AF_INET, ...)}, [16]) = 5 (中略) read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 (中略) write(5, HTTP/1.1 200 OKrnDate: Sun, 10 A..., 2218) = 2218 // ここまでが一つのリクエスト/レスポンス // keepaliveしてないならここで close
  • 32. DEMO #2 - strace web app
  • 33. N[Šs2˜vzsº“Ž Twiggy, Twiggy::Preforkvuevent-drivenv´–Ï (epoll_wait) multi socket listencr—ÔzStarlet (select)
  • 34. Web¡ÖézV›Ì($) lPkPzÝ „Ðzó sþUwv— HTTP鮧¸Çž*]— _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Æ–¾z^|logz„Y‹etc) /_zw‰•zÈèÝXN— HTTPë¸Üò¸ž4e
  • 35. ¶öÜ£òÇ - MySQL socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10 fcntl(10, F_SETFL, O_RDONLY) = 0 fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(10, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr(10.0.0.1 fcntl(10, F_SETFL, O_RDWR) = 0 poll([{fd=10, events=POLLIN|POLLPRI}], 1, 4000) = 1 ([{fd=10, revents=POLLIN} setsockopt(10, SOL_SOCKET, SO_RCVTIMEO, 200334110000000setsockopt(10, SOL_SOCKET, SO_SNDTIMEO, 200334110000000setsockopt(10, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0 setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
  • 36. ¶öÜ£òÇ - memcached socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 fcntl(8, F_GETFL) = 0x2 (flags O_RDWR) fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(8, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr(10.0.0.1 poll([{fd=8, events=POLLOUT}], 1, 250) = 1 ([{fd=8, revents=POLLOUT}]) getsockopt(8, SOL_SOCKET, SO_ERROR, [-511598409602301952], [4]) = 0 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 sendmsg(8, {msg_name(0)=NULL, msg_iov(6)=[{set, 3}, { csrf_token:, 20}, read(8, 0x59f9f50, 1536) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=8, events=POLLIN}], 1, 300) = 1 ([{fd=8, revents=POLLIN}]) read(8, STOREDrn, 1536)
  • 37. DEMO #3 - strace web app with DB and memcached
  • 38. Öì¯èß{­oktV–w ”WvP •d®§éžDxPcr— ­ork”–cache misscr— Ï7wretrycr— TCP ConnectionžPŠ›grvP TCP ConnectionmŸtˆ˜rvP
  • 39. öº #1 - memcached retry problem
  • 40. +180ms tN—APIzë¸Üò¸X|éé–¸ w180msËB 50ms or diev•4x[•P¶vvPtP]vPëØê
  • 41. ?_ Cache::Memcached::FastzèÂЖá·ä–ê (Ò1 XNokzs)
  • 42. M[èo strace -p $pid -e 'trace=sendmsg' (Cache::Memcached::FastXsendmsgžžRz{—orPk)
  • 43. Ë sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
  • 44. ̾|•c èÂЖá·ä–êwretryAÿ (maxs3x) XâorPk Cache::Memcached::Fastzget{cache misstiz)z§ è–žæ*sYvP cache missz_w3xretrycrk retry¹{20msec 20 msec × 3 (retry) × 3 = 180 msec izhit rate{uRvŸƒtPRø{N–ppŽget,{ retrycvP7cžâ˜rwE
  • 45. Ë - with timestamp 22:45:34.517703 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.537967 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.558936 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.579275 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.587428 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.608296 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.628673 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.649146 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.135129 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.155516 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.176027 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.196453 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
  • 46. stracecr‹—t´ö¿ ÙžpWvPÖêwc—
  • 47. ÙžpWvPÖê strace tcpdump gdb
  • 48. Ùžp[Žz zM6 ¡Öézì¯ Ôw{¡Öéz²–È
  • 49. öº #2 - accept(2) thundering herd
  • 50. What is thundering herd The thundering herd problem occurs when a large number of processes waiting for an event are awoken when that event occurs, but only one process is able to proceed at a time. - Wikipedia *[zÖ캸X§zÇé«–s•Ôw”YceX|ö Uz{v{´zÖ캸scW훘vP (Šk{í› ˜—…Y) Ôwé¼–¸XÏ7wv—þU ”[N—z{Æ–¾zremote cacheXexpireckÔw´D wDB„z®§éX—|tW
  • 51. accept(2) and prefork server DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j ˜Õì® Linux kernelX”cvwqueueingcr[˜—zs|鮧 ¸ÇXakÔwOYÖX—Ö캸{´pl] (ˆæ) http://d.hatena.ne.jp/naoya/20070311/1173629378
  • 52. accept(2) and event-driven/ multi-process server DÖ캸X•Ôw´pzsocket fdžepoll_waitsÁ listen socketXreadIÿ = acceptXIÿwvokÔFs epoll_waitcr—ÀrzÖ캸XOYÖX— ´pzÖ캸Xacceptw´ )zÖ캸{óorî (EAGAIN)
  • 53. accept(2) thundering herd accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s Pf loadXÖX— vmstatsf—t... ustsytcsz–Xg–ÖX— (50, 50, 100000tW) Ö캸zcontext switch TeZXOÀWt­›˜—
  • 54. DEMO #3 - EAGAIN and again and again
  • 55. wEn iŽiŽXevent-drivenvŸlciŸvwR£wÖ캸 N^v[rPP”y
  • 56. vv,w”–100Ö캸[•Pw (¼{300[•PN ok)
  • 57. 7• xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr P— Monoceros‹kPwaccepte—Ö캸t鮧¸Çž {ve—Ö캸žñÀw ]—tWƒ event-driventÝêÀÖ캸{øPgXuP with Perl IO::AIO (AnyEvent::IOzbackend) tWŽÇt¶ò¯êÖ ìº¸¼völok–...
  • 58. Štwâ–Še
  • 59. stracez›—Pt_™ ß¿váQXVVP (ø˜˜|uRwsŽv—X) kŠwO_—,zÌÎw{¹PrvP (Devel::KYTProftW sì¯cck[X”Pƒ¥
  • 60. stracez”Pt_™ ̼XÓPs|Ö캸wattache—l]sef˜ — Web¡Öé°–¶æòzDÓmX W—
  • 61. stracelt W•vP_t ¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v ¡Öé°–¶æòszÏ7loop, deep recursion µzuP¡ê³é¹ßtW -d NYTProf Çèò¸Ü–Çä’AszþU tcpdumptWtsharktWngreptWžPŠc“R
  • 62. straces W—_t 鮧¸Ç*]r|ë¸Üò¸4eŠszÔì– Æ–¾¡®º¸zöA öUwDxmemcachedW•getcr—W|DxDB‹w Por—W öUwcachea˜rP—data öUwPícrP—SELECT iz)|ŽP z6crPvP“”
  • 63. H |k–¼l]ustrace{¸ÿzÖêdvP žT|ÚàW•ä—tWiŸv_tŽvP sŽ|žP[X WorP˜|V7€­¦ Web¡ÖézDÓmX W—
  • 64. ˆæ` The Linux Programing Interface ŠPc|K¨„€h(÷§cWvP]u...)Vee à˜§hŽN—]u1000Ù–·Ý|pŠ–„€s{v[ ï1 À󭌫Pvc|éÔ ëò¸tcrŽ6q äz˜{Veel]u|±¿­Œz{ޏXûPzs pRzLinuxÖì¯èÞò¯ tWz[X?|{PPWŽc ˜vP
  • 65. ™G (?) Web+DBz2014ñ10KÿwMÌ(Perl Hackers Hub)ž„Wg rá[_twv–Šck
  • 66. `s×N–XtR`bPŠc k

×