Are you one of those "gifted debuggers" that everyone turns to when they need to solve a difficult problem? Great! This talk isn't for you. For the rest of us, debugging is often considered a mysterious trait that some engineers were born with, but alas, some simply haven't. This talk is here to bust that myth. It calls "bullshit" on the gifted-debugger myth and claims that with well-structured methodology and a couple of simple tips, we can all master debugging and stop using trial and error (and other witchcraft tactics) to find the cause of our problems. This methodology has served me well over the years to solve difficult problems, and will hopefully serve you as well.

1. The Art of Debugging
Avishai Ish-Shalom
github.com/avishai-ish-shalom@nukembergavishai.is@wix.com

2. The Art of Debugging
Avishai Ish-Shalom
github.com/avishai-ish-shalom@nukembergavishai.is@wix.com

3. Randomly rebooting stuff
until it works
01

4. Please don’t
▪ Randomly reboot stuff
▪ Randomly look at metrics
▪ Blindly believe the monitor/metrics
▪ Give the loudest screamer all the attention
▪ Overemphasize the importance of latest data

5. Correlation does not mean causation
▪ Some problems are transient
▪ Some causes are transient, but the system doesn’t recover
▪ Especially when EVERYONE is doing things together

6. ● 1 out of 1000 drivers is drunk
● Breathalyzer detects all drunks but has 5%
false positives
● Drivers stopped at random
A Driver was pulled over, breathalyzer shows
he’s drunk.
What’s the probability he’s really drunk?

7. If you answered 0.95, you have
fallen for the
THE BASE RATE FALLACY
Correct answer: ~ 0.02

8. In a random sample of 1000 drivers, 1 would be drunk
and 49.95 (999 x 0.05) would falsely test as drunk
Base rate of false positives (P(D)=50/1000) >> rate of
drunk drivers (P(drunk)=1/1000)
Explanation

10. Errors are “normal”
▪ Any big enough system has statistical errors/variations
▪ If you randomly search for errors, you will find some somewhere!
▪ And they will probably be unrelated
▪ The base rate of errors >> rate of incidents

11. Without theory,
data is useless and
misleading

12. Debugging 101
02

13. The methodology:
0. Triage
1. Define and narrow down the symptoms
2. Build a (mental) model of the system
3. Deconstruct, create/revise a theory
4. Corroborate the theory
5. Reconstruct and validate
6. Rinse and Repeat

14. 0. Triage
▪ What is the business impact?
▪ Is it actually a problem?
▪ Should we handle this?
▪ Should we handle this now?

15. 1. Define and narrow down the symptoms
▪ Can you recreate the issue?
▪ Isolate the offending conditions
▪ Operational definition of the problem
When? Where? to Whom? Under Which conditions?
E.g.: HTTP GET /bla -> returns 500, from US only with any cookie, user-agent, etc.

16. Examples
■ GET /bla -> returns 500 from all countries, for any headers, in 5% of cases
■ p99 of transaction X is over 500ms for all time intervals since 1 hour ago,
should be under 100ms
■ Transaction Y should return for user XXX returns empty records, should return
100 records

17. 2. The Mental Model
▪ We have one for anything we interact with
▪ Implicit assumptions about how things work
▪ Sometimes wrong
We need to make the model explicit!

18. The Mental Model (example)
How does a oven work?
■ Temperature knob
■ Timer knob

19. The Mental Model (example)
Motorcycle
counter-steering

20. When things don’t make sense
3 options:
▪ We are missing data
▪ The data we have is wrong
▪ Our mental model is wrong

21. 3. Deconstruct, create/revise a theory
▪ Disassemble the system to sub-systems
▪ How are they connected? what is the input/output of each one?
▪ Which sub-system(s) is the cause of the problem?
▪ Or maybe the connection is the problem?
Draw a diagram!

22. 4. Corroborate the theory
▪ Define what metrics/experiments you need to prove/disprove
the theory*
▪ Define the expected results
▪ Get the data
* More on that later

23. Confirmation bias ahead!

24. Confirmation bias
“Confirmation bias is the tendency to search for, interpret,
favor, and recall information in a way that confirms one's
preexisting beliefs or hypotheses”
Always decide what to check before you see the results!

25. 5. Reconstruct and validate
▪ Validate system invariants (e.g. Little’s Law)
▪ Compare the data to expected results
▪ Reconstruct the system from sub-systems
▪ Remember the integration points!

26. 6. Rinse and repeat
▪ Problem found? excellent
▪ Problem narrowed down to sub-system? you are now debugging it. Go
back to #1
▪ Problem not found? revise your theory/model, go back to #2

27. Knowing what to
look at
03

28. What are we looking for?
▪ Should we look for outliers? high values? low values?
▪ Averages? sums? percentiles?
▪ Error counts?

29. What are we looking at?
▪ How was this measured? Client side? Server side?
▪ What aggregation method?
▪ Metrics artifacts?

30. Choose your metric!
▪ Looking for outliers? - percentiles, max, most deviants
▪ Utilization? Totals? - averages, sums
▪ Errors? - error counters
▪ Baselines? - compare with previous timeframe, average, median
▪ Trends? - moving averages (low pass filters)

31. Looking at latency
▪ Percentiles for effect on clients
▪ Sums/Percentiles for used resources
▪ Averages for Little’s Law and the USE method
▪ Bounds (timeouts, max, min)
▪ Always check throughput, frequency

32. Looking at errors
▪ Compare with baseline
▪ Compare with normal throughput
▪ Statistical error? -> is there a statistically significant difference?

33. High cardinality data
▪ Top K
▪ Most deviant
▪ Unique counts

34. The USE method
Utilization - RPM, % busy
Saturation - Queue length, load shedded
Errors - error count
The USE method - Brendan Gregg

35. Q&A
github.com/avishai-ish-shalom@nukembergavishai.is@wix.com

36. Bonus round (a war story)

37. OMG, it’s broken!

38. When in doubt, reboot
▪ Recurring issue, service rebooted instead of debugged
▪ “Known” issue, but nobody understood why

39. Amazing correlation!

40. Once again, with methodology
▪ All of the errors were “business errors”, unrelated.
▪ Use the profiler Luke!
▪ Analyzed results
▪ Construct theory

41. Getting data

42. case class DomainDTO(domainName: String, redirectDomain: Option[String], siteGuid:
Option[UUID])
@tailrec
private def isConnectedToASite(userDomains: List[DomainDTO])(domain: DomainDTO):
Boolean = domain.redirectDomain match {
case Some(redirectToDomain) =>
val redirectToInfoOpt = userDomains.find(_.domainName == redirectToDomain)
redirectToInfoOpt match {
case Some(redirectToInfo) => isConnectedToASite(userDomains)(redirectToInfo)
case None => false
}
case None => domain.siteGuid.isDefined
}

43. But but but….
▪ We don’t have redirect loops!
▪ And even if we did, why didn’t CPU released after timeout?
We needed to revise our mental model!

44. Validating the theory
▪ Queried 400k records
▪ Wrote program that traversed redirects and detects for loops
▪ Redirect loops found! (~30)

45. Reproducing the error

46. Recap
▪ The methodology proved itself (again)
▪ Correlation is not causation
▪ In data we trust (unless we have a good reason not too)

47. Thank You
github.com/avishai-ish-shalom@nukembergavishai.is@wix.com