Your SlideShare is downloading. ×
0
How to Debug
Anything
@jamesgolick
well ok, not anything, but most stuff on
unixy operating systems that have
the tooling ...
https://packagecloud.io
“better than google”
- PC Weekly
Everything is
Terrible
Everything is
Broken
“Correct Code”™
“If you want to deploy high
quality software that performs,
you should expect to fix bugs at
every level.”
- me
“I don’t understand
how this is possible.”
- every programmer ever
0. php
a blind debugging session
the website is down
what we have to work with
• The source code. (nope)
• Knowledge of the system. (nope)
• Familiarity with the programming l...
logging in the real world
(often useless)
#cool
find a pid
sudo strace -ff -s 2048 -p 22935
write(1, "hin", 3) = 3
function name arguments return value
how to read strace output
man 2 write
learn more about your favorite system calls
{sa_family=AF_INET, sin_port=htons(50318), sin_addr=inet_addr("192.168.212.2")}, [16]) = 12	
fcntl(12, F_GETFD) = 0	
fcntl...
writev(12, [{"HTTP/1.0 500 Internal
Server Err"..., 256},
{"37213100000003", 10},
{"30", 2}, {"00000000",
8}], 4) = 276
fin...
open("/var/www/db.in.php", O_RDONLY)	
=	
-1 ENOENT (No such file or directory)
find the cause
hopefully.
open("/var/www/index.php", O_RDONLY)	
=	
13
find the offender
prove your hypothesis
find the offender
fix the bug!
#cool
“I don’t understand
how this is possible.”
- every programmer ever
0. Forget everything you think you know.
1. Get a third party opinion.
third parties
i have known and loved
source: http://www.brendangregg.com/linuxperf.html
1. apt
building a cloud for packages is hard
sudo apt-get update
sudo strace -ff apt-get update
write(1, "Ign http://
192.168.212.136:3000 trusty
Releasen", 62) = 62
find failure
always work backwards
read(6, "400 URI FailurenURI:
https://packagecloud-repositories-
dev2.s3.amazonaws.com/1/1/ubuntu/
dists/trusty/Release?
A...
confirm your hypothesis
#cool
apt-get source apt
read(6, "400 URI FailurenURI: https://
packagecloud-repositories-
dev2.s3.amazonaws.com/1/1/ubuntu/dists/
trusty/Release?
...
locate a hook
stare at the code
confirm your hypothesis
> Content-Type:
> Content-Type: text/
plain
#cool
2. Locate the correct source code.
3. Identify a hard-coded string to grep for.
4. Stare at the code until it makes sense.
5. Fix whatever is broken.
How to Debug Anything
!
!
0. Forget everything you think you know.
1. Get a third party opinion.
2. Locate the correct sou...
Questions?
@jamesgolick
https://packagecloud.io
Upcoming SlideShare
Loading in...5
×

How to Debug Anything

14,709

Published on

Does your code work? Probably not. The libraries you're using probably don't work either. If you're lucky, the OS does, but even then you'll probably find something wrong if you look hard enough.

Debugging is the reason that the last 20% of shipping a product usually accounts for 80% of the time. And yet, there are a million blog posts and talks about writing code, but very few about figuring out why it doesn't work right once you have.

So, how do you find bugs? In this talk I'll explore a set of tools and techniques that have helped me diagnose defects in everything from php code to malloc implementations.

One time I even used this strategy to diagnose an outage in a codebase I'd never seen that was written in a language I barely knew and a framework I'd never heard of - in less than 5 minutes. You'll walk away with this talk with everything you need to learn how to debug anything.

Video: https://www.youtube.com/watch?v=VV7b7fs4VI8

Published in: Software, Technology
0 Comments
78 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
14,709
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
156
Comments
0
Likes
78
Embeds 0
No embeds

No notes for slide

Transcript of "How to Debug Anything"

  1. 1. How to Debug Anything @jamesgolick well ok, not anything, but most stuff on unixy operating systems that have the tooling i’m going to talk about today
  2. 2. https://packagecloud.io “better than google” - PC Weekly
  3. 3. Everything is Terrible
  4. 4. Everything is Broken
  5. 5. “Correct Code”™
  6. 6. “If you want to deploy high quality software that performs, you should expect to fix bugs at every level.” - me
  7. 7. “I don’t understand how this is possible.” - every programmer ever
  8. 8. 0. php a blind debugging session
  9. 9. the website is down
  10. 10. what we have to work with • The source code. (nope) • Knowledge of the system. (nope) • Familiarity with the programming language. (nope) • SSH Access. (yup)
  11. 11. logging in the real world (often useless)
  12. 12. #cool
  13. 13. find a pid
  14. 14. sudo strace -ff -s 2048 -p 22935
  15. 15. write(1, "hin", 3) = 3 function name arguments return value how to read strace output
  16. 16. man 2 write learn more about your favorite system calls
  17. 17. {sa_family=AF_INET, sin_port=htons(50318), sin_addr=inet_addr("192.168.212.2")}, [16]) = 12 fcntl(12, F_GETFD) = 0 fcntl(12, F_SETFD, FD_CLOEXEC) = 0 getsockname(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.168.212.182")}, [16]) = 0 fcntl(12, F_GETFL) = 0x2 (flags O_RDWR) fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0 read(12, "GET / HTTP/1.1rnHost: localhost:8181rnConnection: keep-alivernCache-Control: max-age=0rnAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8rnUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36rnAccept-Encoding: gzip,deflate,sdchrnAccept-Language: en-US,en;q=0.8rnCookie: _jsuid=3899596137; _packages_session=d2NrM3RZMUJXRE8zcFB0aXNGVk83Ny9lRDR3Y09uSVNoRUcrREV0VnF2UjFxRjk1NjAyUzZ5ZG81M1JyczRzUU10ZTBqMXI5QkJXZzFqZnM1RUNmdEdGYmN2eG92SUsvU24wOWhJSlhNZzYrQXdYN2tMYnRZaEhWN3ArbEpiZVpMUWNjWHNRWHc2VjkwQzZ2S 0Y4aGlLeks3MmhoTXBXN2NRWUEwbGFFekpENHdveCtTNXl1MllDUTFzUzZMSU5WZlRqUlQ1aXB2bWVsZDVGVFE1Tlp0UT09LS1vYWdoMk9mZHUvS3U5OWpoME1ZY3pBPT0%3D--8e4ac5c1aebe1e9226063c3d2b83b4176535377arnrn", 8000) = 792 stat("/var/www/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 rt_sigaction(SIGPROF, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0 umask(077) = 022 umask(022) = 077 getcwd("/", 4095) = 2 chdir("/var/www") = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={30, 0}}, NULL) = 0 lstat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 open("/var/www/index.php", O_RDONLY) = 13 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0 mmap(NULL, 447, PROT_READ, MAP_SHARED, 13, 0) = 0x7f8a8d34f000 munmap(0x7f8a8d34f000, 447) = 0 close(13) = 0 getcwd("/var/www", 4096) = 9 lstat("/var/www/./oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/usr/share/php/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) lstat("/var/www/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory) getcwd("/var/www", 4096) = 9 lstat("/var/www/./oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/usr/share/php/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) lstat("/var/www/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory) getcwd("/var/www", 4096) = 9 lstat("/var/www/oh-fuck.php", 0x7fff113446e0) = -1 ENOENT (No such file or directory) open("/var/www/oh-fuck.php", O_RDONLY) = -1 ENOENT (No such file or directory) chdir("/") = 0 umask(022) = 022 open("/dev/urandom", O_RDONLY) = 13 read(13, "33260300377gK222d", 8) = 8 close(13) = 0 open("/dev/urandom", O_RDONLY) = 13 read(13, "427417x3516336260", 8) = 8 close(13) = 0 open("/dev/urandom", O_RDONLY) = 13 read(13, "&M330225-P340345", 8) = 8 close(13) = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a8d34e000 writev(12, [{"HTTP/1.0 500 Internal Server ErrorrnDate: Sat, 14 Jun 2014 17:40:32 GMTrnServer: Apache/2.2.22 (Ubuntu)rnX-Powered-By: PHP/5.3.10-1ubuntu3.11rnVary: Accept-EncodingrnContent-Encoding: gziprnContent-Length: 20rnConnection: closernContent-Type: text/htmlrnrn", 256}, {"37213100000003", 10}, {"30", 2}, {"00000000", 8}], 4) = 276 write(7, "192.168.212.2 - - [14/Jun/2014:17:40:32 +0000] "GET / HTTP/1.1" 500 276 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/ 537.36"n", 199) = 199 times({tms_utime=6, tms_stime=56, tms_cutime=0, tms_cstime=0}) = 1718165756 shutdown(12, 1 /* send */) = 0 poll([{fd=12, events=POLLIN}], 1, 2000) = 1 ([{fd=12, revents=POLLIN|POLLHUP}]) read(12, "", 512) = 0 close(12) = 0 read(4, 0x7fff113490cf, 1) = -1 EAGAIN (Resource temporarily unavailable) ETOOMUCHOUTPUT
  18. 18. writev(12, [{"HTTP/1.0 500 Internal Server Err"..., 256}, {"37213100000003", 10}, {"30", 2}, {"00000000", 8}], 4) = 276 find failure always work backwards
  19. 19. open("/var/www/db.in.php", O_RDONLY) = -1 ENOENT (No such file or directory) find the cause hopefully.
  20. 20. open("/var/www/index.php", O_RDONLY) = 13 find the offender
  21. 21. prove your hypothesis
  22. 22. find the offender
  23. 23. fix the bug!
  24. 24. #cool
  25. 25. “I don’t understand how this is possible.” - every programmer ever
  26. 26. 0. Forget everything you think you know.
  27. 27. 1. Get a third party opinion.
  28. 28. third parties i have known and loved source: http://www.brendangregg.com/linuxperf.html
  29. 29. 1. apt building a cloud for packages is hard
  30. 30. sudo apt-get update
  31. 31. sudo strace -ff apt-get update
  32. 32. write(1, "Ign http:// 192.168.212.136:3000 trusty Releasen", 62) = 62 find failure always work backwards
  33. 33. read(6, "400 URI FailurenURI: https://packagecloud-repositories- dev2.s3.amazonaws.com/1/1/ubuntu/ dists/trusty/Release? AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Si gnature=s/c0fzVQhxpBPbpyTIzCxAfo/ 8g=&Expires=1402837136nMessage: Bad header line nn", 64000) = 230 find the cause
  34. 34. confirm your hypothesis
  35. 35. #cool
  36. 36. apt-get source apt
  37. 37. read(6, "400 URI FailurenURI: https:// packagecloud-repositories- dev2.s3.amazonaws.com/1/1/ubuntu/dists/ trusty/Release? AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Sig nature=s/c0fzVQhxpBPbpyTIzCxAfo/ 8g=&Expires=1402837136nMessage: Bad header line nn", 64000) = 230 locate a hook
  38. 38. locate a hook
  39. 39. stare at the code
  40. 40. confirm your hypothesis
  41. 41. > Content-Type:
  42. 42. > Content-Type: text/ plain
  43. 43. #cool
  44. 44. 2. Locate the correct source code.
  45. 45. 3. Identify a hard-coded string to grep for.
  46. 46. 4. Stare at the code until it makes sense.
  47. 47. 5. Fix whatever is broken.
  48. 48. How to Debug Anything ! ! 0. Forget everything you think you know. 1. Get a third party opinion. 2. Locate the correct source code. 3. Identify a hard-coded string to grep for. 4. Stare at the code until it makes sense. 5. Fix whatever is broken.
  49. 49. Questions? @jamesgolick https://packagecloud.io
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×