Your SlideShare is downloading. ×
How to Debug Anything
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

How to Debug Anything

12,927
views

Published on

Does your code work? Probably not. The libraries you're using probably don't work either. If you're lucky, the OS does, but even then you'll probably find something wrong if you look hard …

Does your code work? Probably not. The libraries you're using probably don't work either. If you're lucky, the OS does, but even then you'll probably find something wrong if you look hard enough.

Debugging is the reason that the last 20% of shipping a product usually accounts for 80% of the time. And yet, there are a million blog posts and talks about writing code, but very few about figuring out why it doesn't work right once you have.

So, how do you find bugs? In this talk I'll explore a set of tools and techniques that have helped me diagnose defects in everything from php code to malloc implementations.

One time I even used this strategy to diagnose an outage in a codebase I'd never seen that was written in a language I barely knew and a framework I'd never heard of - in less than 5 minutes. You'll walk away with this talk with everything you need to learn how to debug anything.

Video: https://www.youtube.com/watch?v=VV7b7fs4VI8

Published in: Software, Technology

0 Comments
78 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
12,927
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
151
Comments
0
Likes
78
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. How to Debug Anything @jamesgolick well ok, not anything, but most stuff on unixy operating systems that have the tooling i’m going to talk about today
  • 2. https://packagecloud.io “better than google” - PC Weekly
  • 3. Everything is Terrible
  • 4. Everything is Broken
  • 5. “Correct Code”™
  • 6. “If you want to deploy high quality software that performs, you should expect to fix bugs at every level.” - me
  • 7. “I don’t understand how this is possible.” - every programmer ever
  • 8. 0. php a blind debugging session
  • 9. the website is down
  • 10. what we have to work with • The source code. (nope) • Knowledge of the system. (nope) • Familiarity with the programming language. (nope) • SSH Access. (yup)
  • 11. logging in the real world (often useless)
  • 12. #cool
  • 13. find a pid
  • 14. sudo strace -ff -s 2048 -p 22935
  • 15. write(1, "hin", 3) = 3 function name arguments return value how to read strace output
  • 16. man 2 write learn more about your favorite system calls
  • 17. {sa_family=AF_INET, sin_port=htons(50318), sin_addr=inet_addr("192.168.212.2")}, [16]) = 12 fcntl(12, F_GETFD) = 0 fcntl(12, F_SETFD, FD_CLOEXEC) = 0 getsockname(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.168.212.182")}, [16]) = 0 fcntl(12, F_GETFL) = 0x2 (flags O_RDWR) fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0 read(12, "GET / HTTP/1.1rnHost: localhost:8181rnConnection: keep-alivernCache-Control: max-age=0rnAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8rnUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36rnAccept-Encoding: gzip,deflate,sdchrnAccept-Language: en-US,en;q=0.8rnCookie: _jsuid=3899596137; _packages_session=d2NrM3RZMUJXRE8zcFB0aXNGVk83Ny9lRDR3Y09uSVNoRUcrREV0VnF2UjFxRjk1NjAyUzZ5ZG81M1JyczRzUU10ZTBqMXI5QkJXZzFqZnM1RUNmdEdGYmN2eG92SUsvU24wOWhJSlhNZzYrQXdYN2tMYnRZaEhWN3ArbEpiZVpMUWNjWHNRWHc2VjkwQzZ2S 0Y4aGlLeks3MmhoTXBXN2NRWUEwbGFFekpENHdveCtTNXl1MllDUTFzUzZMSU5WZlRqUlQ1aXB2bWVsZDVGVFE1Tlp0UT09LS1vYWdoMk9mZHUvS3U5OWpoME1ZY3pBPT0%3D--8e4ac5c1aebe1e9226063c3d2b83b4176535377arnrn", 8000) = 792 stat("/var/www/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 rt_sigaction(SIGPROF, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0 umask(077) = 022 umask(022) = 077 getcwd("/", 4095) = 2 chdir("/var/www") = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={30, 0}}, NULL) = 0 lstat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 open("/var/www/index.php", O_RDONLY) = 13 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 mmap(NULL, 447, PROT_READ, MAP_SHARED, 13, 0) = 0x7f8a8d34f000 munmap(0x7f8a8d34f000, 447) = 0 close(13) = 0 getcwd("/var/www", 4096) = 9 lstat("/var/www/./oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/usr/share/php/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/var/www/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) getcwd("/var/www", 4096) = 9 lstat("/var/www/./oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/usr/share/php/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/var/www/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) getcwd("/var/www", 4096) = 9 lstat("/var/www/oh-fuck.php", 0x7fff113446e0) = -1 ENOENT (No such file or directory) open("/var/www/oh-fuck.php", O_RDONLY) = -1 ENOENT (No such file or directory) chdir("/") = 0 umask(022) = 022 open("/dev/urandom", O_RDONLY) = 13 read(13, "33260300377gK222d", 8) = 8 close(13) = 0 open("/dev/urandom", O_RDONLY) = 13 read(13, "427417x3516336260", 8) = 8 close(13) = 0 open("/dev/urandom", O_RDONLY) = 13 read(13, "&M330225-P340345", 8) = 8 close(13) = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a8d34e000 writev(12, [{"HTTP/1.0 500 Internal Server ErrorrnDate: Sat, 14 Jun 2014 17:40:32 GMTrnServer: Apache/2.2.22 (Ubuntu)rnX-Powered-By: PHP/5.3.10-1ubuntu3.11rnVary: Accept-EncodingrnContent-Encoding: gziprnContent-Length: 20rnConnection: closernContent-Type: text/htmlrnrn", 256}, {"37213100000003", 10}, {"30", 2}, {"00000000", 8}], 4) = 276 write(7, "192.168.212.2 - - [14/Jun/2014:17:40:32 +0000] "GET / HTTP/1.1" 500 276 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/ 537.36"n", 199) = 199 times({tms_utime=6, tms_stime=56, tms_cutime=0, tms_cstime=0}) = 1718165756 shutdown(12, 1 /* send */) = 0 poll([{fd=12, events=POLLIN}], 1, 2000) = 1 ([{fd=12, revents=POLLIN|POLLHUP}]) read(12, "", 512) = 0 close(12) = 0 read(4, 0x7fff113490cf, 1) = -1 EAGAIN (Resource temporarily unavailable) ETOOMUCHOUTPUT
  • 18. writev(12, [{"HTTP/1.0 500 Internal Server Err"..., 256}, {"37213100000003", 10}, {"30", 2}, {"00000000", 8}], 4) = 276 find failure always work backwards
  • 19. open("/var/www/db.in.php", O_RDONLY) = -1 ENOENT (No such file or directory) find the cause hopefully.
  • 20. open("/var/www/index.php", O_RDONLY) = 13 find the offender
  • 21. prove your hypothesis
  • 22. find the offender
  • 23. fix the bug!
  • 24. #cool
  • 25. “I don’t understand how this is possible.” - every programmer ever
  • 26. 0. Forget everything you think you know.
  • 27. 1. Get a third party opinion.
  • 28. third parties i have known and loved source: http://www.brendangregg.com/linuxperf.html
  • 29. 1. apt building a cloud for packages is hard
  • 30. sudo apt-get update
  • 31. sudo strace -ff apt-get update
  • 32. write(1, "Ign http:// 192.168.212.136:3000 trusty Releasen", 62) = 62 find failure always work backwards
  • 33. read(6, "400 URI FailurenURI: https://packagecloud-repositories- dev2.s3.amazonaws.com/1/1/ubuntu/ dists/trusty/Release? AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Si gnature=s/c0fzVQhxpBPbpyTIzCxAfo/ 8g=&Expires=1402837136nMessage: Bad header line nn", 64000) = 230 find the cause
  • 34. confirm your hypothesis
  • 35. #cool
  • 36. apt-get source apt
  • 37. read(6, "400 URI FailurenURI: https:// packagecloud-repositories- dev2.s3.amazonaws.com/1/1/ubuntu/dists/ trusty/Release? AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Sig nature=s/c0fzVQhxpBPbpyTIzCxAfo/ 8g=&Expires=1402837136nMessage: Bad header line nn", 64000) = 230 locate a hook
  • 38. locate a hook
  • 39. stare at the code
  • 40. confirm your hypothesis
  • 41. > Content-Type:
  • 42. > Content-Type: text/ plain
  • 43. #cool
  • 44. 2. Locate the correct source code.
  • 45. 3. Identify a hard-coded string to grep for.
  • 46. 4. Stare at the code until it makes sense.
  • 47. 5. Fix whatever is broken.
  • 48. How to Debug Anything ! ! 0. Forget everything you think you know. 1. Get a third party opinion. 2. Locate the correct source code. 3. Identify a hard-coded string to grep for. 4. Stare at the code until it makes sense. 5. Fix whatever is broken.
  • 49. Questions? @jamesgolick https://packagecloud.io