Preventive Debugging

Poul-Henning Kamp
phk@FreeBSD.org
phk@Varnish.org
@bsdphk
1981
Debugging n.
Removing a BUG, either by tinkering with the
program or by amending the program specification so that th...
Varnish:

A debugging nightmare

* 40.000 threads
* 1 TB common datastructures
* 1.000.000 requests per second
* 30 Gbit/s...
1949
As soon as we started programming, we found to
our surprise that it wasn't as easy to get
programs right as we had th...
1974
Debugging is twice as hard as writing the code
in the first place. Therefore, if you write the
code as cleverly as po...
The crucial debugging insight:
Q: Why are you debugging to begin with ?
A: Because you are not a perfect programmer
Todays task:

Optimize this:

void
debug(input)
{
do {
idea = think(input);
if (idea != NO_IDEA)
patch(idea);
input += tes...
”Option” doesn't mean its optional
If you don't use -Wall -Werror by default,
your code sucks by default.
Also consider:
-...
Get a second, third & fourth opinion
* Use multiple compilers (LLVM vs. GCC)
* lint(1)
* FlexeLint ($1k)
* Coverity
* LLVM...
Random FlexeLint example:
if (!(dfu_root->flags |= DFU_IFF_DFU))
main.c 483 Info 820: Boolean test of a parenthesized assi...
1979
The most effective debugging tool is still careful
thought, coupled with judiciously placed print
statements.
- Brian...
2013
It's even easier to let the program just tell you
where the bugs are.
- Poul-Henning Kamp
Proactively eliminate doubt
#include <assert.h>
if (*q != '0' && r == e) {
if (b != vep->tag) {
l = e - b;
assert(l < size...
Performance price List
●
●
●
●
●
●
●
●

char *p += 5;
strlen(p);
memcpy(p, q, l);
Locking
System Call
Context Switch
Disk ...
What does assert() actually do ?
#define assert(e) ((e) ? (void)0 : __assert( 
__func__, __FILE__, __LINE__, #e))
void
__a...
Make your own asserts
#define AZ(foo)
do {assert((foo) ==
#define AN(foo)
do {assert((foo) !=
#define XXXAZ(foo) do {xxxas...
If it can't happen, assert that
AZ(pipe(vwe->pipes));
AZ(shutdown(sock, SHUT_WR));
AZ(close(fd));
AZ(fstatvfs(fd, &fsst));...
Assert the locking situation
static void
ban_reload(const uint8_t *ban, unsigned len)
{
struct ban *b, *b2;
int duplicate ...
Be constructively paranoid
static enum req_fsm_nxt
cnt_deliver(struct worker *wrk, struct req *req)
{
char time_str[30];

...
(Un)trusted types
struct http {
unsigned
#define HTTP_MAGIC

magic;
0x6428b5c9

struct http_conn {
unsigned
#define HTTP_C...
Miniobj.h
#define ALLOC_OBJ(to, type_magic)
do {
(to) = calloc(sizeof *(to), 1);
if ((to) != NULL)
(to)->magic = (type_mag...
A more helpful assert() handler

Panic from VCL:
PANIC: Had Panic header: fetch
thread = (cache-worker)
ident = FreeBSD,10...
... even more helpful assert() handler
busyobj = 0x803d3a020 {
ws = 0x803d3a098 {
id = "bo",
{s,f,r,e} = {0x803d3bf98,+128...
How ?
Use pthread_setspecific(3) to tie state to thread
In your __assert() function:
Use pthread_getspecific(3) to get tha...
Test that your code works
Automate running of your test-cases
I mean, you do have test-cases ?

right ?

Right ?!
Varnish testing
Varnishtest(1) tool (5000 LOC)
Interprets ”VTC” test-language:
varnishtest "Does anything get through at a...
Varnish testing
336 VTC testcases in 11 categories
Important category:

Regression tests for bugs

”make check” runs test-...
Preventive Debugging Summary
* Code defensively
* Use tools to improve your code & coding
* Assert() that you know what's ...
Does that work ?
Yes!
Varnish delivers a LOT of web-pages
We get approx 1 crash report every 2-3 weeks
We almost never nee...
1946
Intet tab bør ramme os, som kan undgås
ved rettidig omhu.
- Skibsreder A.P. Møller
Upcoming SlideShare
Loading in …5
×

Debugging 2013- Poul henning-kamp

234 views

Published on

Debugging- for rigtige programmører

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
234
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Debugging 2013- Poul henning-kamp

  1. 1. Preventive Debugging Poul-Henning Kamp phk@FreeBSD.org phk@Varnish.org @bsdphk
  2. 2. 1981 Debugging n. Removing a BUG, either by tinkering with the program or by amending the program specification so that the side effect of the bug is published as a desirable feature. See also: KLUDGE; ONE-LINE PATCH; STEPWISE REFINEMENT. - Stan Kelley-Bootle The Devil's D.P. Dictionary
  3. 3. Varnish: A debugging nightmare * 40.000 threads * 1 TB common datastructures * 1.000.000 requests per second * 30 Gbit/s traffic
  4. 4. 1949 As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs. - Maurice Wilkes
  5. 5. 1974 Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Brian W. Kernighan and P. J. Plauger The Elements of Programming Style
  6. 6. The crucial debugging insight: Q: Why are you debugging to begin with ? A: Because you are not a perfect programmer
  7. 7. Todays task: Optimize this: void debug(input) { do { idea = think(input); if (idea != NO_IDEA) patch(idea); input += test(); } while (unhappy(input)); }
  8. 8. ”Option” doesn't mean its optional If you don't use -Wall -Werror by default, your code sucks by default. Also consider: -Wstrict-prototypes -Wpointer-arith -Wcast-qual -Wswitch -Wcast-align -Wchar-subscripts -Wnested-externs -Wformat -Wno-missing-field-initializers -Wmissing-prototypes -Wreturn-type -Wwrite-strings -Wshadow -Wunused-parameter -Winline -Wredundant-decls -Wextra -Wno-sign-compare
  9. 9. Get a second, third & fourth opinion * Use multiple compilers (LLVM vs. GCC) * lint(1) * FlexeLint ($1k) * Coverity * LLVM's static analysis tools * Different CPU/endianess/word-size
  10. 10. Random FlexeLint example: if (!(dfu_root->flags |= DFU_IFF_DFU)) main.c 483 Info 820: Boolean test of a parenthesized assignment main.c 483 Info 774: Boolean within 'if' always evaluates to False [Reference: file main.c: line 483] main.c 483 Info 831: Reference cited in prior message
  11. 11. 1979 The most effective debugging tool is still careful thought, coupled with judiciously placed print statements. - Brian W. Kernighan Unix for Beginners
  12. 12. 2013 It's even easier to let the program just tell you where the bugs are. - Poul-Henning Kamp
  13. 13. Proactively eliminate doubt #include <assert.h> if (*q != '0' && r == e) { if (b != vep->tag) { l = e - b; assert(l < sizeof vep->tag); memmove(vep->tag, b, l); vep->tag_i = l; } return (NULL); }
  14. 14. Performance price List ● ● ● ● ● ● ● ● char *p += 5; strlen(p); memcpy(p, q, l); Locking System Call Context Switch Disk Access Filesystem 10-9s CPU Memory Protection 10-1s Mechanical
  15. 15. What does assert() actually do ? #define assert(e) ((e) ? (void)0 : __assert( __func__, __FILE__, __LINE__, #e)) void __assert(const char *func, const char *file, int line, const char *failedexpr) { } (void)fprintf(stderr, "Assertion failed: ” ”(%s), function %s, file %s, line %d.n", failedexpr, func, file, line); abort(); /* NOTREACHED */
  16. 16. Make your own asserts #define AZ(foo) do {assert((foo) == #define AN(foo) do {assert((foo) != #define XXXAZ(foo) do {xxxassert((foo) #define XXXAN(foo) do {xxxassert((foo) #define WRONG(expl) [...] #define INCOMPLETE(expl) [...] #define Lck_AssertHeld() [...] void WS_Assert(...); void MPL_Assert_Sane(...); void VTCP_Assert(...); ... 0);} while (0) 0);} while (0) == 0);} while (0) != 0);} while (0)
  17. 17. If it can't happen, assert that AZ(pipe(vwe->pipes)); AZ(shutdown(sock, SHUT_WR)); AZ(close(fd)); AZ(fstatvfs(fd, &fsst)); AZ(pthread_mutex_lock(&vsm_mtx)); ... XXXAZ(unlink(vp->fname)); XXXAZ(setgid(mgt_param.gid)); ... sto = calloc(sizeof *sto, 1); XXXAN(sto); ...
  18. 18. Assert the locking situation static void ban_reload(const uint8_t *ban, unsigned len) { struct ban *b, *b2; int duplicate = 0; double t0, t1, t2 = 9e99; ASSERT_CLI(); Lck_AssertHeld(&ban_mtx);
  19. 19. Be constructively paranoid static enum req_fsm_nxt cnt_deliver(struct worker *wrk, struct req *req) { char time_str[30]; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); CHECK_OBJ_NOTNULL(req->obj, OBJECT_MAGIC); CHECK_OBJ_NOTNULL(req->obj->objcore, OBJCORE_MAGIC); CHECK_OBJ_NOTNULL(req->obj->objcore->objhead, OBJHEAD_MAG CHECK_OBJ_NOTNULL(req->vcl, VCL_CONF_MAGIC); assert(WRW_IsReleased(wrk)); assert(req->obj->objcore->refcnt > 0);
  20. 20. (Un)trusted types struct http { unsigned #define HTTP_MAGIC magic; 0x6428b5c9 struct http_conn { unsigned #define HTTP_CONN_MAGIC magic; 0x3e19edd1 struct wrk_accept { unsigned #define WRK_ACCEPT_MAGIC magic; 0x8c4b4d59 struct objcore { unsigned #define OBJCORE_MAGIC magic; 0x4d301302
  21. 21. Miniobj.h #define ALLOC_OBJ(to, type_magic) do { (to) = calloc(sizeof *(to), 1); if ((to) != NULL) (to)->magic = (type_magic); } while (0) #define FREE_OBJ(to) do { (to)->magic = (0); free(to); } while (0) #define CHECK_OBJ_NOTNULL(ptr, type_magic) do { assert((ptr) != NULL); assert((ptr)->magic == type_magic); } while (0)
  22. 22. A more helpful assert() handler Panic from VCL: PANIC: Had Panic header: fetch thread = (cache-worker) ident = FreeBSD,10.0-ALPHA4,amd64,-smalloc,-smalloc,-hcritbi Backtrace: 0x44afeb: PAN_Init+3fb 0x806e05ef0: _end+80673e5b8 0x806c0201d: _end+80653a6e5 0x45f458: VCL_recv_method+7e8 0x4608f5: VCL_backend_response_method+1b5 0x42d3c2: VBF_Fetch+1e32 0x42c01b: VBF_Fetch+a8b 0x44ddd0: Pool_Work_Thread+500 0x474798: WRK_thread+1d8 0x4745ef: WRK_thread+2f
  23. 23. ... even more helpful assert() handler busyobj = 0x803d3a020 { ws = 0x803d3a098 { id = "bo", {s,f,r,e} = {0x803d3bf98,+128,0x0,+57480}, }, do_stream bodystatus = 4 (length), }, http[bereq] = { ws = 0x803d3a098[bo] "GET", "/foo", "HTTP/1.1", "X-Forwarded-For: 127.0.0.1", "Accept-Encoding: gzip", "X-Varnish: 1004", "Host: 127.0.0.1", }, http[beresp] = { ws = 0x803d3a098[bo] "HTTP/1.1", "200", "Ok", "Panic: fetch", "Content-Length: 7", }, ws = 0x803d3a218 { id = "(null)", {s,f,r,e} = {0x0,0x0,0x0,0x0}, }, }
  24. 24. How ? Use pthread_setspecific(3) to tie state to thread In your __assert() function: Use pthread_getspecific(3) to get that state See also backtrace(3) API (OS/Compiler dependent.)
  25. 25. Test that your code works Automate running of your test-cases I mean, you do have test-cases ? right ? Right ?!
  26. 26. Varnish testing Varnishtest(1) tool (5000 LOC) Interprets ”VTC” test-language: varnishtest "Does anything get through at all ?" server s1 { rxreq txresp -body "012345n" } -start varnish v1 -vcl+backend {} -start client c1 { txreq -url "/" rxresp expect resp.status == 200 } -run varnish v1 -expect n_object == 1 varnish v1 -expect sess_conn == 1
  27. 27. Varnish testing 336 VTC testcases in 11 categories Important category: Regression tests for bugs ”make check” runs test-cases (~ 10 minutes) Jenkins tinderbox builds/tests ~10 platforms gcov(1) used to monitor test-coverage. (89%!)
  28. 28. Preventive Debugging Summary * Code defensively * Use tools to improve your code & coding * Assert() that you know what's going on * Fail ASAP. * Dump useful info while you can * Know that your code works & can be executed
  29. 29. Does that work ? Yes! Varnish delivers a LOT of web-pages We get approx 1 crash report every 2-3 weeks We almost never need gdb(1)
  30. 30. 1946 Intet tab bør ramme os, som kan undgås ved rettidig omhu. - Skibsreder A.P. Møller

×