Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sysdig

1,949 views

Published on

Sysdig is a new dynamic tracer for Linux, inspired by strace, dtrace, and tcpdump. Very useful as a super fast strace replacement and systemwide performance/security/etc. diagnostics.

Published in: Technology
  • Be the first to comment

Sysdig

  1. 1. Grzegorz Nosek,
  2. 2. Garbage In, Garbage Out ?
  3. 3. Garbage In, Garbage Out syscalls syscalls *may contain traces of signals and shared memory
  4. 4. Syscalls ssize_t read(int fd, void *buf, size_t count); ssize_t write(int fd, const void *buf, size_t count); int open(const char *pathname, int flags, mode_t mode); int close(int fd); int stat(const char *path, struct stat *buf); ... ! $ grep -c __NR_ /usr/include/asm/unistd_64.h 313 ! $ man 2 read
  5. 5. strace # strace cat /etc/hostname execve("/bin/cat", ["cat", "/etc/hostname"], ... brk(0) = 0x1675000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT mmap(NULL, 8192, PROT_READ|PROT_WRITE, ... access("/etc/ld.so.preload", R_OK) = -1 ENOENT open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=48577, ...}) = 0 mmap(NULL, 48577, PROT_READ, MAP_PRIVATE, 3, 0) = ... close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (...)
  6. 6. that’s cool, but… 1 m illion sysca lls, as fast as possible worst case for a ny tracer # dd if=/dev/zero of=/dev/null bs=1k count=1M 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB) copied, 0.332905 s, 3.2 GB/s # strace -o /dev/null !! 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB) copied, 18.2365 s, 58.9 MB/s 50x overhead
  7. 7. @brendangregg’s http://www.slideshare.net/brendangregg/linux-performance-tools-2014
  8. 8. @brendangregg’s http://www.slideshare.net/brendangregg/linux-performance-tools-2014
  9. 9. hello, sysdig # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules)
  10. 10. hello, sysdig # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) event number, timestamp
  11. 11. hello, sysdig # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) CPU number, process name, pid
  12. 12. hello, sysdig # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) event direction, event type
  13. 13. hello, sysdig enter event # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) exit event syscall
  14. 14. hello, sysdig # sysdig | head -5 3 15:26:36.552482922 0 sysdig (7311) > switch next=329(systemd-udevd) pgft_maj=6 pgft_min=1432 vm_size=26740 vm_rss=3052 vm_swap=0 4 15:26:36.552502349 0 systemd-udevd (329) < read res=2352 data=# This file is part of systemd..#.# systemd is free software; you can redistri 5 15:26:36.552590722 0 systemd-udevd (329) > read fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) size=4096 6 15:26:36.552593880 0 systemd-udevd (329) < read res=0 data= 7 15:26:36.552596220 0 systemd-udevd (329) > close fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules) arbitrary event attributes
  15. 15. events # sysdig -L (...) > open() < open(FD fd, FSPATH name, FLAGS32 flags, UINT32 mode) > close(FD fd) < close(ERRNO res) > read(FD fd, UINT32 size) < read(ERRNO res, BYTEBUF data) > write(FD fd, UINT32 size) < write(ERRNO res, BYTEBUF data) (...)
  16. 16. dd kernel 1073741824 bytes (1.1 GB) copied, 0.332905 s, 3.2 GB/s
  17. 17. dd kernel strace 1073741824 bytes (1.1 GB) copied, 18.2365 s, 58.9 MB/s
  18. 18. dd kernel sysdig ring buffer 1073741824 bytes (1.1 GB) copied, 1.30029 s, 826 MB/s
  19. 19. filters fd.name FD full name. If the fd is a file, this field contains the full path. If the FD is a socket, this field contain the connection tuple. ! proc.apid the pid of one of the process ancestors. ! evt.latency delta between an exit event and the correspondent enter event. ! (...) ! # sysdig -l | grep -Ec '^[a-z0-9_.]+' 88
  20. 20. filters # sysdig fd.name contains shadow 2303 17:30:34.645573185 0 cat (24012) < open fd=-13(EACCES) name=/etc/shadow flags=1(O_RDONLY) mode=0 ! # sysdig evt.res = EACCES or evt.res = EPERM 617 17:32:16.197820784 0 cat (24027) < open fd=-13(EACCES) name=/etc/shadow flags=1(O_RDONLY) mode=0 4333 17:32:26.239052264 0 killall (24028) < kill res=-1(EPERM)
  21. 21. back to that dd again… # sysdig proc.name=not_dd > /dev/null & dd if=/dev/ zero of=/dev/null bs=1k count=1M ; killall sysdig [1] 24070 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB) copied, 0.981408 s, 1.1 GB/s
  22. 22. output formatting sa m e as filters (mostly) # sysdig -p '%user.name %proc.name %fd.name: %evt.res' evt.failed = true ubuntu cat /etc/shadow: EACCES ubuntu cat /usr/share/locale/en_US.UTF-8/LC_MESSAGES/ libc.mo: ENOENT ubuntu cat /usr/share/locale/en_US.utf8/LC_MESSAGES/ libc.mo: ENOENT ubuntu cat /usr/share/locale/en_US/LC_MESSAGES/ libc.mo: ENOENT
  23. 23. bottleneck in a haystack # sysdig -p '%evt.latency.s.%evt.latency.ns %evt.dir %evt.type %fd.name' fd.type contains ip and fd.sport != 22 (...) 0.000000000 >sendto 192.168.1.118:36220->46.28.247.84:80 0.000114365 <sendto 192.168.1.118:36220->46.28.247.84:80 0.000000000 >recvfrom 192.168.1.118:36220->46.28.247.84:80 0.000005090 <recvfrom 192.168.1.118:36220->46.28.247.84:80 0.000000000 >close 192.168.1.118:36220->46.28.247.84:80 0.000001587 <close 192.168.1.118:36220->46.28.247.84:80
  24. 24. sysdig -w sysdig -r sysdig -r sysdig -r .scap file shit’s on fire, yo capture trace file, restore service analyze trace at your leisure
  25. 25. lies, damn lies and benchmarks sysdig -w sysdig -r sysdig -r sysdig -r .scap file do a single benchmark run analyze/postprocess lots of ways
  26. 26. chisels: higher level of awesome Lua sysdig -cl sysdig -i chisel_name sysdig -c chisel_name [args…]
  27. 27. chisel all the things! # sysdig -cl | grep -c ^[a-z] 37 # find /usr/share/sysdig/chisels/ -name '*.lua' | wc -l 42 the extra ones a re utilities to use in ch isels (json, A NSI term ina l, etc.)
  28. 28. chisels: performance bottlenecks Slowest system calls fileslower Trace slow file I/O netlower Trace slow network I/O proc_exec_time Show process execution time scallslower Trace slow syscalls topscalls Top system calls by number of calls topscalls_time Top system calls by time yu p, a ty po ;)
  29. 29. chisels: security list_login_shells List the login shell IDs ! shellshock_detect print shellshock attacks ! spy_users Display interactive user activity power corru pts, absolute power is even more fun
  30. 30. All right gentlemen, we need some system info lsof, ps, n etstat lsof, ps, netstat with time travel http://draios.com/ps-lsof-netstat-time-travel/
  31. 31. gotcha!
  32. 32. version 0.1.91 do you feel lucky? • some syscalls not yet implemented (no args) • it did crash once (fixed immediately though) • PID namespaces ignored • root/privileged user only • one sysdig process at a time way better tha n strace though

×