Download to read offline
Uploaded byBert Jan Schrijver
Debugging distributed systems
This document outlines a structured approach to debugging distributed systems. It begins with observing and documenting the problem. The next steps involve creating a minimal reproducer, debugging the client and server sides, and checking DNS, routing, and network connections. Traffic and messages should also be inspected. The process concludes by wrapping up findings and conducting a post-mortem analysis. Key challenges in distributed systems like concurrency, lack of a global clock, and independent failures are discussed.
Debugging distributed systems
- 1. bertjan@openvalue.eu Debugging distributed systems BertJan Schrijver @bjschrijver
- 2. Debugging distributed systems:the good parts bertjan@openvalue.eu Bert Jan Schrijver @bjschrijver Networking 101 How the internet works
- 3.
- 4. Bert Jan Schrijver Le t ’ s m e e t @bjschrijver
- 5. Why are distributed systemsdifficult? Networking 101 What? Why? ✅ Demo War stories Conclusion W h a t ‘ s n e x t ? Outline A structured approach @bjschrijver
- 6. What is adistributed system?
- 7. A distributed systemis a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another.
- 8. • Concurrency ofcomponents • Lack of a global clock • Independent failure of components • Distributed systems are harder to reason about Characteristics of distributed systems Source: http://www.nasa.gov/images/content/218652main_STOCC_FS_img_lg.jpg
- 9. Working with distributedsystems is fundamentally different from writing software on a single computer - and the main difference is that there are lots of new and exciting ways for things to go wrong. - Martin Kleppmann “ ” Photo: Dave Lehl
- 10. What could possiblygo wrong? “ ” Photo: Dave Lehl
- 11.
- 12. .. in yourbrowser’s address bar and press Enter What happens when you type google.com… Source: https://github.com/alex/what-happens-when
- 13.
- 14.
- 15. Step 1: Observe& document • What do you know about the problem? • Inspect logging, errors, metrics, tracing • Draw the path from source to target - what’s in between? Focus on details! • Document what you know • Can we reproduce in a test? • By injecting errors, for example Tools Whiteboard, documentation, logging, metrics, tracing (opentracing.io), tests, Jepsen
- 16. Step 1: Observe& document
- 17. Step 2: Createminimal reproducer • Goal: maximise the amount of debugging cycles • Focus on short development iterations / feedback loops • Get close to the action! Tools IDE, Shell scripts, SSH tunnels, Curl
- 18. Step 3: Debugclient side • Focus on eliminating anything that could be wrong on the client side • Are we connecting to the right host? • Do we send the right message? • Do we receive a response? • Not much different from local debugging Tools IDE, debugger, logging
- 19. Step 4: CheckDNS & routing • DNS: • Make sure you know what IP address the hostname should resolve to • Verify that this actually happens at the client • Routing: • Verify you can reach the target machine Tools host, nslookup, dig, whois, ping, traceroute, nslookup.io, dnschecker.org
- 20. Step 5: Checkconnection • Can we connect to the port? • If not, do we get a REJECT or a DROP? • Does the connection open and stay open? • Are we talking TLS? • What is the connection speed between us? Tools telnet, nc, curl, iperf
- 21. Step 6: Inspecttraffic / messages • Do we send the right request? • Do we receive the right response? • How do we know? • How do we handle TLS? • Are there any load balancers or proxies in between? Tools curl, wireshark, tcpdump, network tab in browser, mitm/tls proxy
- 22. Step 7: Debugserver side • Inspect the remote host • Can we attach a remote debugger? • See https://youtube.com/OpenValue • Profiling • Strace Tools SSH tunnels, remote debugger, profiler, strace
- 23. Step 8: Wrapup & post mortem • Document the issue: • Timeline • What did we see? • Why did it happen? • What was the impact? • How did we find out? • What did we do to mitigate and fix? • What should we do to prevent repetition? Tools Whiteboard, documentation
- 24. If you reallywant a reliable system, you have to understand what its failure modes are. You have to actually have witnessed it misbehaving. - Jason Cahoon “ ”
- 25. Distributed systems warstories
- 26. The time whereit worked half of the time…
- 27. The one ata school…
- 29. The one withbreaking news…
- 30. Summary: a structuredapproach to debugging distributed systems @bjschrijver Check DNS & routing Check connection Debug client side Create minimal reproducer Debug server side Observe & document Wrap up & post mortem Inspect traffic / messages
- 31.
- 32.
- 33. Thanks for yourtime. Got feedback? Tweet it! All pictures belong to their respective authors @bjschrijver