Downloaded 12 times
![Format
Simple format:
<dir>:<ID>:<tag1>.<tag2>...:<arg1>=<val1>,<arg2>=<val2>...
JSON Format:
["<direction>", <ID>, ["tag1", "tag2"...],
[{"arg1":"val1"}, {"arg2":"val2"}...]]](https://image.slidesharecdn.com/traceeverythingwhenapmmeetssysadmins-170208113242/75/Trace-everything-when-APM-meets-SysAdmins-21-2048.jpg)
Uploaded bySysdig
Trace everything, when APM meets SysAdmins
The document discusses Sysdig, an open source system troubleshooting tool with native container support. It allows capturing system events, filtering, and running scripts on them. Sysdig includes tracing capabilities through Sysdig Tracers, which can inject markers into the event stream to trace functions, network requests, and arbitrary code segments with low overhead. Traces can then be analyzed to troubleshoot performance and measure latencies in code and across systems.
Softwareâ—¦
More Related Content
Similar to Trace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdmins
- 1. Jorge Salamero Sanz @bencerillo TraceEverything When APM meets SysAdmins
- 2. How Linux SysAdmins(used to) do (performance) troubleshooting? • uptime • free -m • ps aux • vmstat 1 • netstat -putan • top • tcpdump
- 3.
- 4.
- 5. How developers aredoing (distributed) tracing and profiling? • OpenTracing and alternatives: • Zipkin, Dapper, HTrace, X-Trace • Commercial: • New Relic, AppDynamics, Dyn, SPM • Self-brewed and hacks: • statsd, JMX • print • logs can bite you in the ass
- 6. What if wehad something... • Open source • Simple and easy to use (trade-off vs eBPF/bcc) • Lightweight • Could work everywhere (including containers)
- 7. Sysdig Open Source systemtroubleshooting with native container support (htop, vmstat, netstat, lsof, tcpdump…) Monitoring, alerting, troubleshooting tool for Docker, Kubernetes, Mesos, RancherOS, GCE, ECS
- 8.
- 9.
- 10.
- 11.
- 12. Sysdig • Capture systemevents, filter and run scripts • Trace dumps for analysis • Container support • CLI or curses interface
- 13.
- 14. Sysdig Tracers Sysdig Tracers: Systemcall tracing
- 15. Sysdig Tracers: Systemcall tracing • Inject markers inside Sysdig event stream • Mark being and end of spans • Function calls • Network request • Arbitrary piece of code • From any language (write to /dev/null) • Low overhead (<1us)
- 16.
- 17.
- 18. Event stream Open Read Close Connect Read Write Read Read Write Close > Request <Request Enter Tracer Exit Tracer
- 19.
- 20. Event stream Open Read Close Connect Read Write Read Read Write Close > Request <Request > Request.Download < Request.Download Parent Span Child Span
- 21. Format Simple format: <dir>:<ID>:<tag1>.<tag2>...:<arg1>=<val1>,<arg2>=<val2>... JSON Format: ["<direction>",<ID>, ["tag1", "tag2"...], [{"arg1":"val1"}, {"arg2":"val2"}...]]
- 22. Example while : do # Starta trace named 'website-latency' echo ">::website-latency::" > /dev/null # Download the sysdig home page curl -s http://sysdig.org > /dev/null # End the the trace echo "<::website-latency::" > /dev/null done
- 23. Fun things youcan do â—Ź Measure latencies in your code â—Ź Save and filter traces with sysdig â—Ź Analyze traces with csysdig â—Ź Inspect system activity inside execution spans â—Ź Trace-aware Log Monitoring â—Ź Export trace latencies using statsd
- 24.
- 25. Further reading â—Ź Documentation: â—‹https://github.com/draios/sysdig/wiki/Tracers â—‹ https://sysdig.com/blog/tracking-down-application-bottlene cks-with-tracers/ â—Ź Integrations â—‹ Python https://github.com/draios/tracer-py â—‹ Node: https://https://github.com/tj/node-trace â—‹ Go: https://github.com/tj/go-trace
- 26.