• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Dtrace и немного магии
 

Dtrace и немного магии

on

  • 452 views

 

Statistics

Views

Total Views
452
Views on SlideShare
452
Embed Views
0

Actions

Likes
3
Downloads
5
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Dtrace и немного магии Dtrace и немного магии Presentation Transcript

    • и немного магии Saturday, September 28, 13
    • Dtrace: что за зверь? • Инструмент • Язык • Образ мыслей • Сообщество Saturday, September 28, 13
    • У нас есть проблемы... • Отладка • Профилирование • Трассировка • Сбор статистики Saturday, September 28, 13
    • И есть инструменты... • Отладка: gdb, mdb • Профилирование: oprofile • Трассировка: truss, strace • Сбор статистики: kstats, statfs Saturday, September 28, 13
    • Много инструментов... • Отладка: gdb, mdb • Профилирование: oprofile • Трассировка: truss, strace • Сбор статистики: kstats, statfs Saturday, September 28, 13
    • • Отладка • Профилирование • Трассировка • Сбор статистики One tool to rule them all Dtrace Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D • Consumers Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D • Consumers • Библиотека Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D • Consumers • Библиотека • Фреймворк Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D • Consumers • Библиотека • Фреймворк • Точки инструментирования Saturday, September 28, 13
    • Dtrace: что за зверь? • Язык D • Consumers • Библиотека • Фреймворк • Точки инструментирования • Провайдеры Saturday, September 28, 13
    • Среда обитания Software stack Saturday, September 28, 13
    • Среда обитания Software stack Saturday, September 28, 13
    • Probes % sudo dtrace -l ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR 4 fbt fcsm fcsm_job_cache_constructor entry 5 fbt fcsm fcsm_job_cache_constructor return 6 fbt fcsm fcsm_job_cache_destructor entry 7 fbt fcsm fcsm_job_cache_destructor return ... 5759 syscall read entry 5760 syscall read return 5761 syscall write entry 5762 syscall write return ... 6518 proc genunix cfork create 6519 sysinfo genunix cfork sysfork 6520 proc genunix proc_exit exit 6521 proc genunix lwp_exit lwp-exit 6522 proc genunix proc_exit lwp-exit Saturday, September 28, 13
    • Probes % sudo dtrace -l ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR 4 fbt fcsm fcsm_job_cache_constructor entry 5 fbt fcsm fcsm_job_cache_constructor return 6 fbt fcsm fcsm_job_cache_destructor entry 7 fbt fcsm fcsm_job_cache_destructor return ... 5759 syscall read entry 5760 syscall read return 5761 syscall write entry 5762 syscall write return ... 6518 proc genunix cfork create 6519 sysinfo genunix cfork sysfork 6520 proc genunix proc_exit exit 6521 proc genunix lwp_exit lwp-exit 6522 proc genunix proc_exit lwp-exit Format: provider:module:function:name Example: proc:genunix:cfork:create Saturday, September 28, 13
    • Язык D Переменная Проба Предикат Действие #!/usr/sbin/dtrace -s BEGIN { ! c = 0; ! printf("How many times does ls call read?"); } syscall::read:entry /execname == "ls"/ { c++; } END { ! printf("t%d times", c); } % ./test.d dtrace: script './test.d' matched 3 probes CPU ID FUNCTION:NAME 0 1 :BEGIN How many times does ls call read? ^C 0 2 :END ! 6 times Переменная Проба Предикат Тело Saturday, September 28, 13
    • Язык D #!/usr/sbin/dtrace -s BEGIN { ! c = 0; ! printf("How many times does ls call read?"); } syscall::read:entry /execname == "ls"/ { c++; } END { ! printf("t%d times", c); } % ./test.d dtrace: script './test.d' matched 3 probes CPU ID FUNCTION:NAME 0 1 :BEGIN How many times does ls call read? ^C 0 2 :END ! 6 times Not thread safe! Saturday, September 28, 13
    • Отладка Saturday, September 28, 13
    • Отладка: волшебный printf • printf(...); • std::cout << ... << std::end; • System.out.println(...); • echo “...”; • log(loglevel, message) Saturday, September 28, 13
    • Отладка: проблема останова • make CFLAGS=-DDEBUG • rmmod; insmode Saturday, September 28, 13
    • • Не останавливаем приложение • Не перекомпилируем • Не запускаем его в дебагере • Подключаемся на лету когда угодно и куда угодно • Почти не теряем в производительности Отладка: dtrace Saturday, September 28, 13
    • Дебажим: кто сказал мяу? Дано: некий демон zpool_worker активно дергает функции библиотеки libzfs. Одна из библиотечных функций, zpool_vdev_attach(), по непонятной причине возвращает ошибку. Задача: Залезть в дебри libzfs и понять откуда именно и почему возвращается ошибка Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Провайдер pid, развернется в pid<ARG> Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Модуль, нету модуля Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Имя пробы Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Thread local variable Saturday, September 28, 13
    • #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Цепляемся ко всем функциям из libzfs Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Выполнить только если: a) Мы пришли из zpool_vdev_attach() b) Код возврата (arg1) == -1 Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s pid$target::zpool_vdev_attach:entry { self->trace = 1; } pid$target:libzfs::return /self->trace && (int)arg1 == -1/ { ustack(); exit(0); } Распечатать стэк и выйти Saturday, September 28, 13
    • Дебажим: кто сказал мяу? % ./who_said_meow.d -p 373 dtrace: script './who_said_meow.d' matched 478 probes CPU ID FUNCTION:NAME 0 52733 zfs_ioctl:return libzfs.so.1`zfs_ioctl+0x2c zpool_worker`do_zpool_attach_or_replace+0x154 zpool_worker`zpool_rpc_attach+0x9f zpool_worker`attach_invoke+0x70 zpool_worker`rpc_invoke+0xbb zpool_worker`rpc_server_loop+0x9d zpool_worker`rpc_worker_main_mode+0xc9 zpool_worker`rpc_worker_main+0x20 zpool_worker`main+0x6c zpool_worker`_start+0x83 Saturday, September 28, 13
    • Дебажим: кто сказал мяу? % dtrace -n 'fbt::zfs_ioc_vdev_attach:return / execname=="zpool_worker"/ {trace(arg1);}' dtrace: description 'fbt::zfs_ioc_vdev_attach:return ' matched 1 probe CPU ID FUNCTION:NAME 0 60612 zfs_ioc_vdev_attach:return 16 Проделаем то же самое в пространстве ядра. Для начала узнаем код ошибки... Saturday, September 28, 13
    • Дебажим: кто сказал мяу? % dtrace -n 'fbt::zfs_ioc_vdev_attach:return / execname=="zpool_worker"/ {trace(arg1);}' dtrace: description 'fbt::zfs_ioc_vdev_attach:return ' matched 1 probe CPU ID FUNCTION:NAME 0 60612 zfs_ioc_vdev_attach:return 16 FBT: провайдер точек инструментирования ядерных функций Saturday, September 28, 13
    • Дебажим: кто сказал мяу? % dtrace -n 'fbt::zfs_ioc_vdev_attach:return / execname=="zpool_worker"/ {trace(arg1);}' dtrace: description 'fbt::zfs_ioc_vdev_attach:return ' matched 1 probe CPU ID FUNCTION:NAME 0 60612 zfs_ioc_vdev_attach:return 16 Запускать пробу только если функцию дернул zpool_worker Saturday, September 28, 13
    • Дебажим: кто сказал мяу? #!/usr/sbin/dtrace -s fbt::zfs_ioc_vdev_attach:entry /execname == "zpool_worker"/ { self->trace = 1; } fbt:zfs::return /self->trace && arg1 == 16/ { stack(); exit(0); } Saturday, September 28, 13
    • Дебажим: кто сказал мяу? % ./who_said_meow_kern.d dtrace: script './who_said_meow_kern.d' matched 2439 probes CPU ID FUNCTION:NAME 0 61288 spa_open_ref:return zfs`spa_open_common+0x279 zfs`spa_open+0x23 zfs`zfs_ioc_vdev_attach+0x32 zfs`zfsdev_ioctl+0x327 genunix`cdev_ioctl+0x45 specfs`spec_ioctl+0x5a genunix`fop_ioctl+0x7b genunix`ioctl+0x18e unix`_sys_sysenter_post_swapgs+0x149 Saturday, September 28, 13
    • Flowindent: что делает redis? #!/usr/sbin/dtrace -s #pragma D option flowindent pid$target:::entry { } pid$target:::return { } Saturday, September 28, 13
    • Flowindent: что делает redis? % ./trace_redis.d -p 17926 ... 0 -> aeProcessEvents 0 -> gettimeofday 0 -> __commpage_gettimeofday 0 -> mach_absolute_time 0 <- mach_absolute_time 0 <- __commpage_gettimeofday 0 <- gettimeofday 0 -> kevent 0 <- kevent 0 -> time 0 -> fegetenv 0 -> bzero$VARIANT$sse42 0 <- bzero$VARIANT$sse42 0 <- fegetenv 0 -> gettimeofday 0 -> __commpage_gettimeofday 0 -> mach_absolute_time 0 <- mach_absolute_time 0 <- __commpage_gettimeofday 0 <- gettimeofday 0 -> fesetenv 0 <- fesetenv 0 <- time 0 -> gettimeofdaySaturday, September 28, 13
    • Статистика Saturday, September 28, 13
    • Статистика • Dtrace - прекрасный инструмент для сбора статистики • Dtrace умеет считать • Dtrace умеет измерять • Dtrace даже умеет визуализировать Saturday, September 28, 13
    • Статистика: quantize #!/usr/sbin/dtrace -s pid$target::malloc:entry { ! @a = quantize(arg0) } Saturday, September 28, 13
    • Статистика: quantize #!/usr/sbin/dtrace -s pid$target::malloc:entry { ! @a = quantize(arg0) } Aggregation Saturday, September 28, 13
    • Статистика: quantize #!/usr/sbin/dtrace -s pid$target::malloc:entry { ! @a = quantize(arg0) } Aggregation: @a.append(arg0) quantize(@a) Saturday, September 28, 13
    • Статистика: quantize % ./showmalloc.d -p 17926 # Redis ... value ------------- Distribution ------------- count 4 | 0 8 |@@@@@@@@ 70 16 |@@@@@@@@@@@@@@@@@@@@@ 187 32 |@@@ 23 64 |@@@@@ 47 128 |@@@ 23 256 | 0 Saturday, September 28, 13
    • Статистика: quantize % ./showmalloc.d -p 17926 # Safari ... value ------------- Distribution ------------- count -1 | 0 0 | 2 1 | 4 2 | 0 4 | 72 8 |@@@@@ 1362 16 |@@@@@@@@@ 2395 32 |@@@@@@ 1604 64 |@@@@@@ 1663 128 |@@@ 840 256 |@@ 401 512 |@@@@ 1128 1024 |@ 295 2048 |@ 252 4096 |@ 150 8192 | 2 16384 | 7 32768 | 32 65536 | 80 131072 | 1 262144 | 0 524288 | 0 Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_saddr] = count(); } Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_saddr] = count(); } Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_saddr] = count(); } struct tcpinfo (TCP header fields) Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_saddr] = count(); } struct ipinfo (IP information) Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_saddr] = count(); } Императивная запись: @pkts[<ip_daddr>]++ Saturday, September 28, 13
    • Кто стучится в порт ко мне? #!/usr/sbin/dtrace -s tcp:::receive /args[4]->tcp_dport == 80/ { @pkts[args[2]->ip_daddr] = count(); } % ./port.d ... 192.168.0.50 34 192.168.0.102 11 192.168.0.12 2 Saturday, September 28, 13
    • Trace disk I/O size #!/usr/sbin/dtrace -s io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount); } Saturday, September 28, 13
    • Trace disk I/O size #!/usr/sbin/dtrace -s io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount); } struct bufinfo (абстракция, описывающая I/O request) Saturday, September 28, 13
    • Trace disk I/O size #!/usr/sbin/dtrace -s io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount); } % ./iostat.d dtrace: description 'io:::start ' matched 1 probe CPU ID FUNCTION:NAME 0 1031 buf_strategy:start 183 cfprefsd 4096 2 1031 buf_strategy:start 183 cfprefsd 4096 0 1031 buf_strategy:start 222 iTerm 24576 0 1031 buf_strategy:start 222 iTerm 4096 0 1031 buf_strategy:start 7098 Keynote 4096 0 1031 buf_strategy:start 7098 Keynote 4096 2 1031 buf_strategy:start 78602 Mail 4096 2 1031 buf_strategy:start 78602 Mail 4096 2 1031 buf_strategy:start 78602 Mail 4096 Saturday, September 28, 13
    • Покажи свои файлы #!/usr/sbin/dtrace -s syscall::open:entry { printf("%s %s", execname, copyinstr(arg0)); } % ./trace_opens.d dtrace: description 'syscall::open*:entry ' matched 4 probes CPU ID FUNCTION:NAME 4 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data 4 141 open:entry ocspd /private/var/db/crls/crlcache.db 5 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/window_142.data 5 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/.DS_Store 6 141 open:entry mds . 6 141 open:entry Safari /Users/dk/Library/Saved Application State/com.apple.Safari.savedState/data.data 6 141 open:entry Safari /Users/dk/Library/Saved Application State/com.apple.Safari.savedState/windows.plist Saturday, September 28, 13
    • Покажи свои файлы #!/usr/sbin/dtrace -s syscall::open:entry { printf("%s %s", execname, copyinstr(arg0)); } % ./trace_opens.d dtrace: description 'syscall::open*:entry ' matched 4 probes CPU ID FUNCTION:NAME 4 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data 4 141 open:entry ocspd /private/var/db/crls/crlcache.db 5 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/window_142.data 5 141 open:entry mdworker /Users/dk/Library/Saved Application State/com.googlecode.iterm2.savedState/.DS_Store 6 141 open:entry mds . 6 141 open:entry Safari /Users/dk/Library/Saved Application State/com.apple.Safari.savedState/data.data 6 141 open:entry Safari /Users/dk/Library/Saved Application State/com.apple.Safari.savedState/windows.plist не забываем, что мы в пространстве ядра Saturday, September 28, 13
    • Профайлим Saturday, September 28, 13
    • Где самые горячие функции? #!/usr/sbin/dtrace -s profile-997 /arg1/ { @a[execname, ufunc(arg1)] = count(); } tick-60s { trunc(@a, 5); exit(0); } провайдер profile Saturday, September 28, 13
    • Где самые горячие функции? #!/usr/sbin/dtrace -s profile-997 /arg1/ { @a[execname, ufunc(arg1)] = count(); } tick-60s { trunc(@a, 5); exit(0); } user mode Saturday, September 28, 13
    • Где самые горячие функции? #!/usr/sbin/dtrace -s profile-997 /arg1/ { @a[execname, ufunc(arg1)] = count(); } tick-60s { trunc(@a, 5); exit(0); } составной ключ Saturday, September 28, 13
    • Где самые горячие функции? #!/usr/sbin/dtrace -s profile-997 /arg1/ { @a[execname, ufunc(arg1)] = count(); } tick-60s { trunc(@a, 5); exit(0); } % ./hotfuncs.d CPU ID FUNCTION:NAME 4 288995 :tick-60s Aquamacs Aquamacs`Fassoc WebProcess JavaScriptCore`JSC::SlotVisitor::copyAndAppend WebProcess JavaScriptCore`JSC::Structure::visitChildren Aquamacs Aquamacs`assq_no_quit Skype libobjc.A.dylib`objc_msgSend Saturday, September 28, 13
    • Волшебный timestamp #!/usr/sbin/dtrace -s nfsv3:::op-write-start { self->write_start_time = timestamp; } nfsv3:::op-write-done { this->diff = timestamp - self->write_start_time; @a1["avg"] = avg(this->diff); @a2["min"] = min(this->diff); @a3["max"] = max(this->diff); @plot = quantize(this->diff); } tick-60s { exit(0); } Saturday, September 28, 13
    • Волшебный timestamp #!/usr/sbin/dtrace -s nfsv3:::op-write-start { self->write_start_time = timestamp; } nfsv3:::op-write-done { this->diff = timestamp - self->write_start_time; @a1["avg"] = avg(this->diff); @a2["min"] = min(this->diff); @a3["max"] = max(this->diff); @plot = quantize(this->diff); } tick-60s { exit(0); } Используем thread local переменную Saturday, September 28, 13
    • Волшебный timestamp #!/usr/sbin/dtrace -s nfsv3:::op-write-start { self->write_start_time = timestamp; } nfsv3:::op-write-done { this->diff = timestamp - self->write_start_time; @a1["avg"] = avg(this->diff); @a2["min"] = min(this->diff); @a3["max"] = max(this->diff); @plot = quantize(this->diff); } tick-60s { exit(0); } Saturday, September 28, 13
    • Волшебный timestamp #!/usr/sbin/dtrace -s nfsv3:::op-write-start { self->write_start_time = timestamp; } nfsv3:::op-write-done { this->diff = timestamp - self->write_start_time; @a1["avg"] = avg(this->diff); @a2["min"] = min(this->diff); @a3["max"] = max(this->diff); @plot = quantize(this->diff); } tick-60s { exit(0); } avg(all diffs in @a1) min(all diffs in @a2) max(all diffs in @a3) Saturday, September 28, 13
    • Волшебный timestamp % ./nfsv3_write_perf.d avg 31157 min 8472 max 19504129 value ------------- Distribution ------------- count 4096 | 0 8192 |@@@@@@@@@@ 2472 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 6610 32768 |@@ 536 65536 | 36 131072 | 1 262144 | 9 524288 | 69 1048576 | 13 2097152 | 3 4194304 | 0 8388608 | 1 16777216 | 0 Saturday, September 28, 13
    • Flamegraphs Saturday, September 28, 13
    • Flamegraphs % dtrace -x ustackframes=100 -n ‘profile-997 /execname == “mysqld”/ {@[ustack()] = count();} tick-60s {exit(0);}’ > out.user_stacks % ./stackcollapse.pl out.user_stacks | ./flamegraph.pl > graph.svg Saturday, September 28, 13
    • Dtrace в разных ОС • OpenSolaris/Illumos • FreeBSD • NetBSD • MacOSX • Oracle Solaris • Oracle Linux? Saturday, September 28, 13
    • What about GNU/Linux We don’t take kindly dtrace guys like you Saturday, September 28, 13
    • Есть ли жизнь вне C? • С++: name mangling • Java: USDT • node.js, perl, ruby: libusdt based providers • libusdt: github.com/chrisa/libusdt/ Saturday, September 28, 13
    • Полезные ссылки • http://www.brendangregg.com/dtrace.html • http://www.dtracebook.com/index.php/Main_Page • https://wiki.freebsd.org/DTrace • https://blogs.oracle.com/rweisner/entry/dtrace_and_c_objects • http://www.joyent.com/blog/bruning-questions-debugging-node-apps-with-dtrace • https://github.com/chrisa/libusdt • http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/ Saturday, September 28, 13
    • Вопросы Saturday, September 28, 13